ueda mep presentation nov 2008. final

MEP Evaluation System: Past, Present and Future- November 2008 NIST MEP

M A N U F A C T U R I N G E X T E N S I O N P A R T N E R S H I P

Manufacturing Extension Partnership: Manufacturing Extension Partnership: Demonstrating Innovative Strategies in Demonstrating Innovative Strategies in Performance MetricsPerformance Metrics

Carroll Thomas, Partnership CatalystCarroll Thomas, Partnership CatalystKenneth P. Voytek, Chief EconomistKenneth P. Voytek, Chief Economist

November 2008 NIST MEP

2


MEP Evaluation System: Past, Present and Future-

Presentation Purpose & Presentation Purpose & ObjectivesObjectives About The Manufacturing Extension Partnership (MEP)

- underlying rationale for the program

How MEP Evaluates Partnership Opportunities

Examine how MEP assesses and evaluates client, center, and program performance


3



“It’s déjà vu all over again!” ~ Yogi Berra


4



Mission Statement“To strengthen the global competitiveness of US-based manufacturing by providing

information, decision support, and implementation of innovative approaches focused

on leveraging technologies, techniques, and business best practices.”

The National Institute of Standard and Technology (NIST) is an agency of the U.S. Commerce Department


5



What/Where is MEP 59 “Centers”1600 Field Staff

440 Service Locations


6



The MEP Program in Short….The MEP Program in Short…. The centers, serving all 50 States and Puerto Rico, are linked together through the

Department of Commerce's National Institute of Standards and Technology. - 59 assistance centers with 442 field locations

- System wide, Non-Federal staff is ~1,600

- Contracting with over 3,000 third party service providers

- Partnership Model – Federal/State/Industry

- MEP System budget ~ $300M

- 1/3 Federal, ~1/3 State and >1/3 Industry (fees for services)

- Emphasis on performance – program and center – measured based upon impact of center services on client firms.

http://www.nist.gov/

http://www.nist.gov/


7



Focus on meeting manufacturer’s short term needs, but in context of overall company strategy

Working directly with over 27,000 manufacturing companies a year*

UNDERLYING IT ALL – Establish and expand partnerships with other like-minded, mission oriented organizations at Federal, state and local level

*Based on FY2007 MEP Center reported performance data.

What MEP Does


8



Client Impacts Resulting from MEP Services

New Sales $3.11 Billion

Retained Sales $3.65 Billion

Capital Investment $1.65 Billion

Cost Savings $1.115 Billion

Jobs Created and Retained 52,585

FY 2006 economic impact results are based on a survey of 4,959 MEP-served establishments out of 5,384 attempted.


9



= Partnership

The Green Suppliers Network, an innovative collaboration between government and industry,

Focus: Address the dual challenge of reducing the environmental impact of small and mid-sized manufacturing suppliers while simultaneously increasing those companies’ efficiency, productivity and profitability.

Approach: Lean & Clean on-site reviews

Value Add: Expands lean definition of waste to include environmental considerations

Deliverable: Measurable business and environmental impacts


10



What to Look for in Potential Partners

Center Impact / Revenue / Reach

Stakeholder Influence, Visibility

Complementary Products &

Services

Referrals (Access to

Clients) & Joint Marketing

Basic Partnership Model

Financial Resources (for

Centers, Clients)

Other Benefits

Buried Treasure/Other Benefits

Types of Partnership

Benefits

• Alignment of missions

• Overlap in customer base or target markets

• Brand prominence, credibility and fit

• Complementary products & services

• Visibility with stakeholders

• Financial resources for clients or the center

• Opportunities to share or leverage resources

Partnership BenefitsWhat to Look for in a Potential Partner (to obtain these benefits)


11



How to Evaluate “ROI” of Potential Partnerships

Partnership Benefits Identified in this Study

Referrals/Cross-Marketing

Complementary Products/Services

Stakeholder Visibility

Financial

Other Benefits❑ Might other benefits result from the

relationship, such as other networking opportunities? Or new techniques?

❑ Does partner offer potential center funding?❑ Enable center cost share? Pay for staff?❑ Offer financial resources for clients?❑ Provide opportunity to share resources? Co-

locate? Reduce duplication?

❑ Does partner provide visibility with state stakeholders?

❑ Does partner offer complementary services?❑ Do their services have the potential to

generate significant revenue and impact?❑ What is the competitive situation for these

services?

❑ How much staff time is required to develop the relationship?

❑ How much training time is required to understand partner responsibilities?

❑ What opportunities must be foregone to avoid competition, turf issues?

❑ Will this partnership dilute or change our brand? Or cause us to stray from our mission?

❑ Will the personal relationship among leadership be supportive or create obstacles?

❑ What is the risk of not engaging with this partner?

❑ Are they business-driven and prompt in responding to client needs?

❑ Does partner serve the same target market?❑ Provide referrals and confer credibility?❑ Get us into companies that we wouldn’t get

into on our own?❑ Is their a good fit between our brands?

Evaluating Potential Return vs. Evaluating Investment & Cost

Return on Investment of Potential Partnerships

❑ Might other benefits result from the relationship, such as other networking opportunities? Or new techniques?

❑ Does partner offer potential center funding?❑ Enable center cost share? Pay for staff?❑ Offer financial resources for clients?❑ Provide opportunity to share resources? Co-

locate? Reduce duplication?

❑ Does partner provide visibility with state stakeholders?

❑ Does partner offer complementary services?❑ Do their services have the potential to

generate significant revenue and impact?❑ What is the competitive situation for these

services?

❑ How much staff time is required to develop the relationship?

❑ How much training time is required to understand partner responsibilities?

❑ What opportunities must be foregone to avoid competition, turf issues?

❑ Will this partnership dilute or change our brand? Or cause us to stray from our mission?

❑ Will the personal relationship among leadership be supportive or create obstacles?

❑ What is the risk of not engaging with this partner?

❑ Are they business-driven and prompt in responding to client needs?

❑ Does partner serve the same target market?❑ Provide referrals and confer credibility?❑ Get us into companies that we wouldn’t get

into on our own?❑ Is their a good fit between our brands?

Evaluating Potential Return vs. Evaluating Investment & Cost

Return on Investment of Potential Partnerships


12



Evaluation and Performance Measurement

Performance Measurement: The ongoing monitoring and reporting of program progress and accomplishments using pre-selected performance measures.

Program evaluation: systematic study using measurement and analysis to answer specific questions about how well a program is working to achieve its outcomes and why.


13



The Changing Face of Manufacturing: The Changing Face of Manufacturing: Getting Smaller; Need to Get SmarterGetting Smaller; Need to Get Smarter

Facts about small manufacturers:

- Represent 99 percent of all manufacturing establishments

- Employ 10.2 million people -- 70 percent of all manufacturing employment, and

- Account for about 57 percent of the total value-added by all U.S. manufacturers

The Challenges for Small Manufacturers:

- In 2002, productivity per employee in large establishments was 69 percent higher than productivity in small establishments. In 1967, productivity per employee was “only” 26 percent higher.

- Over the last 10 years, productivity among large firms grew 7 percentage points faster than small firms.

- Market Failures in several dimensions: firm, inter-firm, consulting/services, public failure.

$20,000

$40,000

$60,000

$80,000

$100,000

$120,000

$140,000

$160,000

$180,000

$200,000

1967 1972 1977 1982 1987 1992 1997 2002

Small Manufacturers (lessthan 500 employees)

Large Manufacturers (500or more employees)


14



Evaluations should assess the extent to which the NIST Evaluations should assess the extent to which the NIST MEP strategy is achieving its goalsMEP strategy is achieving its goals

- The principal goal is to help manufacturers increase The principal goal is to help manufacturers increase the competitiveness and productivity of U.S. the competitiveness and productivity of U.S. manufacturing and NIST MEP has a two-pronged manufacturing and NIST MEP has a two-pronged strategy:strategy:

- Help manufacturers in the United States improve performance – i.e., make products the right way

- Help manufacturers in the United States grow their business – i.e., make the right products for the right customers profitably


15



Performance should be assessed on multiple Performance should be assessed on multiple dimensionsdimensions

Coverage. The extent to which companies in the target population have received services.

Outputs: The volume of services undertaken by affiliated centers as reflected in key outputs.

Operating efficiency. The extent to which processes are efficient, yielding the highest level of output for a given level of input, or alternatively, using the fewest resources to achieve a given level of output.

Financial sustainability. The extent to which affiliated centers have sufficient resources to cover their operating costs and needed investments.

Effectiveness. The extent to which the objectives have been achieved in terms of improved performance of the target population and associated economic outcomes.

Leadership, strategy and operations. The organizational performance of affiliated centers.

Outputs

EfficiencyFinancial

Sustainability

Effectiveness

LEADERSHIP,STRATEGY AND

OPERATIONS

Coverage


16



Effectiveness should focus on the impact of centers on the change in value-added and sales within manufacturing firms.

The crucial issue in assessing impacts is comparing the observed situation with the counterfactual

T i m e

O u t c o m e

O b s e r v e d o u t c o m ew i t h i n t e r v e n t i o n

W h a t w o u l d h a v e h a p p e n e dw i t h o u t i n t e r v e n t i o n

I m p a c t}


17



The Center Balancing Act

Revenue

MarketPenetration

Client Impact

The Program Balancing Act

MarketPenetration

Financial Stability

MarketPenetration

Economic Impact


18



MEP Measures Success & Performance At Different Levels

Client Impacts and Performance Improvements

Center Performance & Impacts

System Performance & Impacts


19



Manufacturing Extension Partnership (MEP) Program: Impact Path

Inputs

Funding• Federal Funding• State/local funding• Client fees

Staff• Trained MEP Centerstaff• National MEP program staff provideprogram oversight,training, technicalbusiness assistance

Activities

MEP Centers provide:

• Information• Decision Support• Implementation assistance• Center’s services help manufacturing clients adoptnew and more advanced manufacturing technologies, techniques, and business practices

Firm-level Business Impacts

• Cost savings• Capital investment• Jobs created• Sales (new and retained)• Profit Margin• Improvements in:

• Manufacturing systems• Human resources system• IT systems• Marketing and sales systems• Management systems

Outcomes

• Productivity growth of smallmanufacturing firms• Increased global competitiveness of US-based manufacturers• Improved supply chain efficiency• Improved job opportunitiesfor US workers• Higher rates of businesssurvival

Output Tracking

MEP tracks the number of clients each year over

20,000) and the total number of activities performed by MEP

Centers (over 30,000/year)

Measuring Client ImpactsThrough an annual client survey, MEP tracks the impacts of Center assistance on several major firm-

level indicators (sales, cost savings, jobs). As a set, these

indicators suggest the presence of business changes that are positively associated with

productivity growth

Program Evaluation

Need more access or creative ways to conduct

such studies

Center Review

Annual operating plan,External reviews,

Center impacts metrics,and quarterly

reporting

Measuring Along the WayMeasuring Along the Way


20



$0

$500

$1,000

$1,500

$2,000

$2,500

$3,000

$3,500

$4,000

$4,500

$5,000

FY 199

7

FY 199

8

FY 199

9

FY 200

0

FY 200

1

FY 200

2

FY 200

3

FY 200

4

FY 200

5

FY 200

6

FY 200

7*

Pro

gra

m Im

pa

ct

($ M

illio

ns

)

New Sales Investments Cost Savings Federal Funding

Project-based survey

Client-based survey

Center Metrics in Place

*FY 2007 data represented only ¾ of impacts

MEP Performance Continues to Increase While Federal Funding Remains Relatively Flat


21



CLIENTS TOUCHED AND CLIENTS TOUCHED AND SURVEYED OVER TIMESURVEYED OVER TIME

1,000

6,000

11,000

16,000

21,000

26,000

31,000

FY 199

9

FY 200

0

FY 200

1

FY 200

2

FY 200

3

FY 200

4

FY 200

5

FY 200

6

FY 200

7

FY 200

8*

Client Interactions

Surveyable Clients

Clients completing survey

Note: Data on clients served did not become a GPRA target until FY 2001. In addition, FY 2000 and FY 2007 data on clients surveyed represent partial years. Surveys are done with a lag.

Client Interaction Metric in effect


22



MEP’s Data Reporting & Center Performance SystemMEP’s Data Reporting & Center Performance System

Operating Plan: Annual plan (linked to the center’s strategic plan) that outlines the anticipated activities and results for the coming year.

Quarterly Data Reporting: Center reports progress and client project data quarterly.

Annual Review: Each year prior to annual renewal of federal funding, the performance of the center is reviewed comprehensively by NIST-MEP or an external panel.

External Panel Review: Every two years the center is reviewed by an external panel that assesses the center performance and alignment with their operational and strategic plans.

Third Party Client Survey: NIST sponsors a national survey conducted by an independent third party that quarterly collects data from center clients on the business impacts of the services provided by their local center. NIST-MEP uses this performance data as a core component in reviewing center performance. The results also provide the centers with a tool to measure their effectiveness and benchmark their performance against other centers.


23



Center Performance MetricsCenter Performance Metrics

Percent quantified impacts (10 points)

Clients Served per Million Federal dollar (15 points)

Investment Leverage Ratio (25 points)

Bottom-line client impact ratio (25 points)

Impacted clients per Million Federal $ (25 points)

Center Performance Metrics

MEP Evaluation System: Past, Present and Future- November 2008 NIST MEP


LOW IMPACT, LOW PENETRATION

HIGH IMPACT, LOW PENETRATION

HIGH IMPACT, HIGH PENETRATION

LOW IMPACT, HIGH PENETRATION

The Variation in Center Performance, Q2 2008


25



Median Center Performance Over Time

48

5.89

6.25

204

50

14.05

9.72

170

64

29.46

23.25

314

0 50 100 150 200 250 300 350

Impacted Clients

Bottom-Line ClientImpact Ratio

InvestmentLeverage Ratio

Clients Served

2001 2004 2008


26



Components of MEP’s Performance & Evaluation Components of MEP’s Performance & Evaluation System: The PortfolioSystem: The Portfolio

Firm Level Studies:

- Impact Data from Third Party Client Impact SurveyImpact Data from Third Party Client Impact Survey: new sales, cost-savings, : new sales, cost-savings, investments attributed to MEP services [GPRA metrics] used in part to develop investments attributed to MEP services [GPRA metrics] used in part to develop center performance metrics. center performance metrics.

- Longitudinal StudiesLongitudinal Studies: focuses on comparing the competitive performance of : focuses on comparing the competitive performance of MEP clients relative to similar firms that did not receive MEP services.MEP clients relative to similar firms that did not receive MEP services.

- Case StudiesCase Studies: focus on successful MEP projects to gain insight into variables : focus on successful MEP projects to gain insight into variables at both the firm and industry-level that impact technology adoption and business at both the firm and industry-level that impact technology adoption and business transformation.transformation.

Impact Studies: regional economic impact analysis of MEP centers


27



Attributes of Current SystemAttributes of Current System

Mix of performance measurement and evaluation approaches

Triangulation of results through different methods and approaches

Focus on intermediate outcomes and ultimate program goals

Multi-level & Multi-dimensional (firm, center, and system)


28



MEP Measurement System is a MEP Measurement System is a Competitive AdvantageCompetitive Advantage

Communicate to federal, state, local stakeholders- Replicable, believable data - Voice of Customer results- Measurement process investment better than any

economic development program

Operational Improvement- Benchmarking against other centers- Measurable impact focus drives service delivery

choices


29



Thanks!Thanks!

Contact the MEPContact the MEP Just call our toll-free number 1-800-MEP 4 MFG (1-800-637-

4634), and your call will be automatically routed to the MEP center that serves your region.

www.mep.nist.gov

Program Questions: mailto:[email protected]

NIST MEP100 Bureau Drive, Stop 4800

Gaithersburg, MD 20899-3460(301) 975-5020

http://www.mep.nist.gov/

mailto:[email protected]

ueda mep presentation nov 2008. final

Business

nist mep

mep program

fy2007 mep center

mep services new sales

technology nist

future focus

program performance

performance program