What Lies Beneath… avoiding the risk of undervaluation

Luke Georghiou

PREST, Manchester Business School

http://www.mbs.ac.uk/PREST

Proposition 1

• Present techniques, and particularly surveys, focus on predictable and linear effects while missing or recording only perfunctorily a series of other effects which may be of equal or greater value.

[Behavioural additionality] can help explain that the effects of policy interventions may be greater than those perceived through a narrower input-output lens, and that the range of effects and the means by which they are achieved are complex mechanisms requiring substantial effort for successful policy design.

Proposition 2

• Even where effects are properly identified and/or calculated there is a tendency to underestimate the contribution of public intervention.

Historical context -co-evolution…

• Evaluation approaches tend to co-evolve cumulatively with policy development, eg– 1970s modification of peer review to extend criteria– 1980s interest began in collaborative R&D programmes– 1990s rise of:

• performance indicators • emphasis on knowledge transfer indicators• Institutional evaluation using programmatic approach

– 2000s interest in;• evaluation of system capabilities eg national or regional

systems• aggregate or interactive effects of policies (“policy mix”)• effect of “soft” policy tools such as foresight• strategic and persistent effects of public support

(“behavioural additionality”)

Evaluation and situating the object in its context

• When we come to evaluate support for industrial R&D (or other research) we encounter key problem

• Project fallacy– Confusion in timing and scope between the unit of

research and the contractual entity– Research impacts are often cumulative over series

of projects– Effects of research policies result from an

interaction between the measure and the strategy of the research performer

Project fallacy

• Key problem of “project fallacy” in which policymaker assumes that a contract is equivalent to a project – In practice contracted work is often only part of a longer and

broader project

Real deliverablesContract Real project

Contract deliverables

EUREKA Initiative as Example

• Evaluation approach since mid-1990s based on “Continuous and Systematic Evaluation” – Final reporting requirements replaced by

short impact survey– Follow-up after one and three years if

market effects reported

EUREKA CSE

Typical question (1)

Typical question (2)

Gives some useful results eg turnover effects - skewed returns

% of turnover accounted by % of firms in EUREKA Initiative

Additional Annual Turnover(achieved)

0%

10%

20%

30%

40%

50%

60%

70%

0=1 1=5 5=25 more

Meuro

% of projects % of turnover

But several problems…

• Inaccurate completion of survey forms– Trivial but important – confusing currency units– Non-trivial and also important - What to measure,

when to measure, how to interpret all dependent upon the respondent’s underlying model of innovation – implicit or explicit

• Large proportion of benefits dealt with only by means of importance rating with no model of how they interact or translate to measurable effects

EUREKA 2006

• Changing methodology – keeping CSE as supplementary approach

• Main effort on panel carrying out case studies of high impact projects – use skew as basis for purposive sample

• Using methodology developed by PREST in evaluation of Japanese National R&D Programme for Medical and Welfare Equipment using Beta method as starting point

Iceberg model

Sales of innovative product

Reduced process costs

Licence income

Firm strategy, organisation and method learningUse of technology in other parts of the businessNew contacts/networks & prestigeEmployment, competence & training

Spillovers to non-participants

User and social benefits

Effects on firm strategy

• Projects normally aligned with firm strategies but also create pathways to transform them – Movement from R&D services/consultancy to

manufacture – Assisting privatised state enterprises to become

commercially viable– Improvements in organisation and methods

induced by the project, including working at a higher technological level frequently internalised by the firms.

Use of technologies in other parts of the business

Neglected aspect of benefits• Opening new technological options through

follow-on projects• Application of technologies developed to

exploit new opportunities, potentially with turnover effects equal to or greater than in the originally intended application.

Contacts, networks and prestige

New contacts and networks generated by projects Mobilising effect on large sectors eg AUTOSAR

network in the automotive industry which emerged from ITEA EAST EEA.

Prestige effects of the EUREKA label helpful to give smaller firms a foothold in the

market and visibility to investors.

Employment

• Quantitative versus qualitative important here– In quantitative treatment need to consider

jobs induced or displaced in value chain– Qualitative can include upgrading of

competences, hiring higher proportion of scientists and engineers

– Also elimination of dangerous or unpleasant work

Spillovers

• Using Jaffe’s classifications:– Knowledge spillovers appear weak and rare

because SMEs have focussed niche strategies and large consortia working in parallel with competitors

– Market spillovers very strong – quick measure is to look at payback time for customers

– Network spillovers usually exploiting complementarities of equipment or equipment/service

User benefits

• Improved environments, health, safety or simply from better value for money and functionality in everyday objects

Bringing it to reality…

• Benefits come as a package and much is lost in disaggregation

• 3 cases

Case study 1- Σ!1692 Sanifogger• Lynx Award for humidifiers that keep food fresh in supermarket

storage and refrigerated cabinets• Contronics turnover from €700k in 1999 to €3.8M in 2005 almost

all from project• Transfer of technology to vehicle air-conditioning cleaner led to

orders for 10k units p.a. (contact to new partner through EUREKA Secretariat)

• Organisational learning included introduction of quality manager in Contronics and market repositioning in UK partner Pendred leading to €375k sales

• 16 jobs created in partners, 10 in sub-contractors • Customer benefit of 50% reduction in wastage with 6 months

payback for fruit/veg and 4 week payback for meat – estimate of €42M p.a. customer benefit

• EUREKA label providing substantial marketing and visibility effects

Case study 2 – Σ!1872 P3D

• Developed passive surveillance systems for tracking aircraft 200km around airports

• ERA is SME privatised 1994 and using EUREKA project in R&D based shift from military to commercial orientation

• ERA and its spin-offs 30-50% of total ERA's sales in period of 2002-4, total value of €5.4-9.0M over 3 years.

• Fourteen new jobs in ERA and 20 in partner firms performing outsourced manufacture

• Chairing international standards setting group• Reduced costs by half compared with conventional

systems• Fundamental contribution to safety

Case study 3 - Σ!2275 Visualix

• Gendex Dental Systems developed powerful digital system for high resolution dental radiography

• Sales, since 2002, worth €28.6M to produce 8,635 units– 20% of firm’s total 2005 sales

• 3 new jobs created, 2 in R&D now developing future projects using extended knowledge base and IPR platform

• Eliminates film development and liquid disposal for dentists giving them 2-4 year payback and more time with patients

• Patient gets immediate image and possibility of better treatment decisions

Additionality – what difference does the intervention make?

• Input additionality – are resources being spent on desired target?

• Output additionality – what proportion of outputs result from particular intervention?

• Behavioural additionality – what difference in behaviour results from the intervention– Concept formulated mid-1990s to help explain consistent

evaluation findings– Rooted in question of how support interacts with strategies

and capabilities of funded organisations– Looks closely at mode of delivery of support for research– Emphasis on persistent changes– OECD project explored measurement issues in industrial

grant schemes

Effects during grant cycle

Awarenessapplication

Contract

Monitoring

ResearchPost-project

support

Failed/non-applicants stimulated IPR &

collaboration rules adopted

Feedback Research direction or linkages changed

Exploitation Route changed

EUREKA contribution to effects of its projects

• Public funding – Role of public funding in enabling, extending or accelerating

the work stressed – No success cases without public funding and some not

returned to EUREKA because funding no longer available– Advice from funding agencies helpful to smaller or less

experienced firms and large firms spoke very positively of the role of cluster offices.

• Collaboration – Complementary skills brought together, often in vertical

structure, linking users and suppliers– In clusters “pre-competitive” configurations, establishing de

facto standards or sharing cost of developing technologies to be applied in different sectors

– Often the initiative of EUREKA that brought together these networks

Under-evaluation = Under-investment

• Our failure to appreciate the full extent of both the private and the social returns to R&D is a key reason for the international decline in government investment in research

• It is possible, though not proven, that industry itself may underestimate its own benefits from R&D, particularly under shareholder pressure to show short term profits

Conclusions (1)• “Standard” questionnaire based evaluation

approaches are never a sufficient means to assess effects however sophisticated an analysis is subsequently applied– Nonetheless they can be a useful first step towards

identifying case studies• High impact cases can be a good approximation

to total effects in an R&D initiative – but not for other instruments where effects are more

evenly spread (eg information and advisory services• Behavioural additionality perspective opens the

way to a much more comprehensive appreciation of effects– Ultimately we are concerned as much with building

capacities as with short-term impacts• Evaluators need to probe what lies beneath the

surface if they are to succeed in capturing and measuring the value created

Conclusions (2)

• Evaluators need to probe what lies beneath the surface if they are to succeed in capturing and measuring the value created

• This allows us to address some of the key policy issues of today