business and systems aligned. business empowered. tm three approaches to measuring the economic...

Business and Systems Aligned. Business Empowered.TM

Three Approaches to Measuring the Economic Impacts of RTD

Toronto, ON.October 29, 2005

Presentation to the CES/AES ConferenceA. Dennis Rank and Frederick Kijeck

© 2005 BearingPoint LP 2

Overview

Three studies of three programs.

Programs had very different goals, structure, operation, and impacts.

Three different challenges, and solutions.

Thus three entirely different methods.

All studies measure benefits from “national perspective”


Three Programs

Precarn Inc. Canadian Spatial Reference System

(CSRS)

High Performance Polymer Membranes

(HPPM)

What National not-for-profit consortium to develop

intelligent systems sector

Infrastructure for 4-D positioning (e.g., latitude, longitude, height, time)

Fundamental research in novel polymer

membranes

How Matching (to 40%) project grants for RTD

NRCan develops & maintains system

Value-added products & services to users

10-year gov’t funding for collaborative industry-led

RTD

Who IS sector only Surveying, also transport, utilities, resource

industries, etc. etc.

Japanese chemical industry as a whole

When 1998 – 2004+ Ongoing. 1981 – 1991

Why Explicit commercial goals

No real commercial goals or activities

Make more competitive (BUT!), strengthen pure

research ability


Three Studies

Precarn CSRS HPPM

Full evaluation of Phase III

Measure economic impacts Phases I – III

ID needs of modern, healthy NSRS

ID 3 options for system requirements & service

delivery

Full evaluation

Assist METI in evaluation system development


Precarn Challenge and Solutions

Straightforward Challenges Solution

Process OK None -

Data Two previous impact studies

P. knew projects & likely impacts

Technology proprietary

Financials confidential

Confidential data collection system

Aggregated reporting only (even to client)

Respondents Easy to find, cooperative (BUT!)

None -

Uses & Users

Easy to pick “big winners”

1 product, 1 firm, 1 market

None -

Method Bottom-up, Rigorous Partial Benefit-Cost (B-C)

Impact Analysis

Some existing sales, cost savings

Had business plans

Evolution of projects & firms over time

Unsure of L-T impacts

Lower & upper bounds

Conservative assumptions

Detailed documentation


CSRS Challenges And Solutions


Process None CSRS highly-complex

Distrust & suspicion

Multiple study “clients”

Never entirely solved

Study team w/ high technical understanding

Many working papers, lines of evidence

Data Large literature on nature &

impacts of NSRS

Quantitative data sparse or non-existent

Uses & Users

Experts known – but mistrusted!

(a) Very large # of uses & users, but many unknown

Many users D/K CSRS

(a) Focus on major, knowledgeable users

(b) Too many “big winner” possibilities – difficult or

impossible to quantify

(b) Dropped “big winner” case study approach

Method (1) Top-down, semi-quantitative, Gross Value Added

Impact Analysis

Lots on qualitative

Lots on need for individual elements

Couldn’t assess quantitative, incremental

value-added

Analysis of elements distrusted

Bolstered by: (2) Qualitative case studies

(3) Similar int’l studies

(4) Benchmarking NSRS elements, economic indices


HPPM Challenges And Solutions


Process Interview on-site, in-person

Working through translator

Bright interpreter OK with technical terms

Data Reasonable records

Study done 12 years later

Exceptional security & confidentiality concerns

Blessing in disguise

Never entirely solved

Uses & Users

Most participants still around, at

same org.

“No impacts of HPPM research at all”

Re-interviewed respondents, pressed for

details

Impacts None Narrow view of impacts

All research fundamental

All impacts were “knock-on” and “spin-off”

Followed-up on very concrete, explicit long-term

indirect & non-linear impacts

Method (1) Qualitative case studies of Knock-on & Spin-off impacts

Impact Analysis

Considerable disagreement re: impacts among

participants

Bolstered by: (2) Modified international peer review

(Bibliometrics considered, but dropped)


Precarn Methodology

Bottom-up Partial B-C: Net benefits = (Gross benefits of “big winners”) – (Implementation costs).

Total program costs = (Grants and contributions) + (Overhead costs) + (Costs to all other parties).

All benefits and costs discounted.

Net Present Value (NPV) NPV = (Net benefits of “big winners”) – (Total program costs to all parties).

Benefit: Cost ratio (B/C) B/C = (Net benefits of “big winners”) ÷ (Total program costs to all

parties).


Precarn Findings

Commercial net benefits, cost savings from 6 Big Winner Projects, @ 5.5% discount rate: Lower bound:

— Net Benefits = $312 million

— NPV = $83.7 million

— B/C ratio = 1.37

Upper bound:

— Net Benefits = $1.3 billion

— NPV = $1.1 billion

— B/C ratio = 5.77


Precarn Lessons Learned

Crucial to know projects and impacts, likely “big winners”.

One technology, one firm, one market always easiest to deal with – makes partial B-C an obvious choice.

But ONLY if can get confidential $ data.

Clients still need “education” about B-C.


CSRS Methodology

Top-down Gross Value Added (GVA):— CSRS influence estimated (Low to High) for 8 sectors & 30+ sub-sectors

— Influence quantified by proxy % index.

— $ Influence per sector = (Sector GVA) x (Proxy % index).

— Total influence of CSRS on Canada = Sum of GVA influences per sector.

Bolstered by other study activities:— User surveys & interviews, focus groups, policy analysis, review of

international and Canadian literature, qualitative case studies, etc.

— Benchmarking to NSRS elements in other countries.

— NSRS comparison indices in benchmark countries on $/capita, $/sq. km., $/GDP.


CSRS Findings

CSRS “influences” $60B - $90B of Canadian GVA, or about 6% – 9% of total.

Backbone of many industrial, government, and research applications, though many users know little about NSRS or CSRS.

This figure (as far as possible) modeled on geodetic impacts alone.


CSRS Lessons Learned

Top-down sometimes only way to go.

But top-down methods difficult to understand, open to interpretation – need other lines of evidence (the more quantitative the better).

ID of incremental value-added is harder than in bottom-up methods.

Entrenched suspicions difficult to overcome.


HPPM Methodology

Qualitative case studies of interesting commercial and policy impacts.

Follow-up case studies focused on very specific, concrete long-term impacts first identified by ex-HPPM expert.

Impacts confirmed through modified peer review re: research quality, relevance, and impacts.


HPPM Findings

HPPM believed to be a commercial failure: No direct commercial applications of technologies (e.g., application of patents).

Partially due to nature of technical targets – HPPM would not support applied R&D, and difficult to commercialize R&D not in partners’ product line.

Scientific importance actually quite high: HPPM showed better to improve existing membranes than develop new ones.

Understood physics and chemistry re. how and why membranes worked.

These were breakthrough concepts for all membranes – not just polymers, & not just separation membranes.


HPPM Findings (cont’d)

High long-term commercial impacts: New products (e.g., nanofilters) related to HPPM topics, or non-polymer

materials (e.g., ceramics), or other types of polymers, or in entirely new fields.

Linked – often quite indirectly – to HPPM ideas.

Some firms world leaders & “first to market” partially because of HPPM.

Impossible to obtain sales figures for these products and processes.

Impossible to judge exact role of HPPM, but good reasons to believe it was at least moderately important in these later developments.

Also impacts on competencies, researchers’ careers, & innovation networks.

Possible impacts on patent laws, R&D policies, & design of R&D programs.


HPPM Lessons Learned

Tracking long-term impacts “easier” when considerable time elapsed.

Need to be very specific and explicit when trying to ID and track these impacts – they are not well understood by participants.

Need external, independent confirmation.

Client and analyst need to be flexible re: methods.


Overall Lessons Learned

1. No “magic bullet” – cookie cutter approach won’t work.

2. Difficult to overcome entrenched attitudes and suspicions – many lines of evidence, lots of documentation help, but don’t entirely solve problem.

3. Very useful for program to track “big winners” & users, even for top-down studies.

4. Multi-step process for long-term knock-on & spin-off benefits works, but need to ID very explicit impacts – and don’t ignore non-technology transfer benefits.

5. Follow your nose, be flexible – don’t ignore oddballs.

business and systems aligned. business empowered. tm three approaches to measuring the economic...

Documents