Econometric evaluation of public innovation subsidies: state of the art, limitations and future research

Dirk CzarnitzkiK.U.Leuven and ZEW Mannheim

Paris, November 9th 2010

Introduction

• R&D is subject to market failure– External effects– Financial constraints

• Governments try to „correct“ market failure using several policies– Governments invest in public science– Intellectual property rights systems– R&D collaborations are exempt from anti-trust policy– Public R&D grants or tax credits for companies

• Often preferred treatment for research consortia

• Recently especially industry science collaborations

Introduction: Direct subsidies

• Problem: crowding-out may occur!– Once subsidies are available, companies have an incentive to

apply for any project (even the privately profitable ones) as subsidy comes at marginal cost equal to zero.

– Subsidies may not only stimulate the projects with high social return.

– In the worst case, private funding is simply replaced with public funding.

– How can government select projects that were not carried out otherwise?

• How to evaluate the „success“ of a policy?

An example of self-assessment by companies (259 subsidized German companies in 2001)

0% 10% 20% 30% 40% 50% 60% 70%

Project implementation became possible

Project start accelerated

Project duration reduced

Project scope extended

Increased technological level

Led to patent application

The Evaluation Problem

• Aim of quantitative methods of evaluation is the measurement of effects generated by policy interventions on certain target variables;

• in innovation context, for example:• impact of R&D subsidies on firms‘ private innovation/R&D

expenditure (input)• or on other variables like patent applications etc. (output)

• David et al. (2000) review the literature on crowding-out effects and Klette et al. (2000) survey microeconometric studies including output analyzes (like firm growth, firm value, patents etc.)

• Critique of early surveys: results varied a lot, possibly because of deficencies of methods that researchers used in the past selection bias (next slide)

• Recent survey, Cerulli (2010) surveys more recent literature that used econometric methods for „treatment effect“ estimation.

The Evaluation Problem

In most cases, one is interested in the average „treatment effect on the treated“ (TT), that is, the difference between the actual observed value of the subsidized firms and the counterfactual situation:

„Which average value of R&D expenditure would the treated firms have shown if they had not been treated“

Problem: The counterfactual situation is never observable and has to be estimated!

| 1 | 1T CTT E Y S E Y S

S:= Status of group, 1 = Treatment group; 0 = Non-treated firms

Outcome: YT = in case of treatment; YC = counterfactual situation

The Evaluation Problem

How can we estimate the counterfactual situation?

Problem: Those firms receiving a treatment may be different from those that don‘t.

Thus: We cannot use a random sample of non-treated without any adjustment.

Example: Agencies that fund R&D follow a picking the winner strategy, as they want to maximize the outcome of the funded projects.

Firms that show high R&D in the past, professional R&D management, good success with their other R&D projects will be preferably selected by policy makers.

Subsidy receipt becomes an endogenous variable (depending on the firms characteristics).

Solution: experimental setting, that is, a random assignment of treatments;or the evaluation of policy via treatment effects estimators in non-experimental settings.

Evidence on treatment effects estimation

• „Buzzwords“:– Econometric Matching– Selection models (also parametric treatment effects models, or

control function approaches)– Instrumental variable estimation– (conditional) difference-in-difference estimation– Regression discontinuity design

• Most recent studies find positive effects of R&D subsidies on R&D investment

• However– Researchers often only observed subsidized yes/no, instead of

amount of funding. – Only some studies that use exact amount of funding/year.

Real effects or wage effects?

• Recent literature goes beyond the mere treatment effect on the treated estimation.

• For instance, „Goolsbee (1998) critique“– One may find increased investment because of subsidy– However: risk that subsidy only ends up in higher wages of

researchers– If wage increase does not coincide with productivity increase,

there would be no „real“ effect on knowledge creation.

• Recent evidence for Europe– Kris Aerts, Ph.D. Thesis 2008 K.U.Leuven, for Flanders– Pierre Mohnen and Boris Lokshin for NL

„real“ R&D is also stimulated by subsidies.

Policy design: large versus small firms

• Researchers often find larger treatment effects for smaller firms than for larger firms

• This may be a statistical artefact to some extent, though– Often only small sample on really large firms (data limitation)– Proportion of subsidized R&D is much smaller in large firms

compared to total R&D budget than in smaller firms• Easier to find a „significant“ effect in small firms than in large firms.

• Measurement or specification problem

• See e.g. Aerts and Czarnitzki (2006), IWT study.

Policy design: size of subsidies

• Some evidence that small subsidies are „not useful“– Gonzales et al. (2006)

• However, more research is needed here.• Could also be a measurement problem as subsidy is

usually related to total R&D of the firm, and thus it is not surprising that small subsidy has no large effects.

• If one applies an estimation technique that is very flexible with regard to functional form assumptions– (GPS method for estimating dose-response functions)

• …we can learn about about potential crowding out effects in different areas of the subsidy distribution

A dose response function-5

05

10

-6 -4 -2 0 2ln(subsidy)

Dose--response function Nadaraya-Watson estimatesyols lnfue

A dose response function0

12

34

5D

ose

--re

sponse

funct

ion/y

ols

/yols

2

-6 -4 -2 0 2ln(subsidy)

Dose--response function yolsyols2

Estimated elasticities0

.51

-6 -4 -2 0 2ln(subsidy)

hy E OLS LINE OLS SQ

Policy design: e.g. collaborative research

• Often preferential treatment of R&D consortia, especially industry science collaborations

• External effects can be internalized within consortia by collaboration

• Duplicate research avoided• Participants can benefit from bundling knowledge and

realizing economies of scope (knowledge from a collaborative project can be used for other projects)

• Thus, there may be a „money effect“ and a „spillover effect“.

Evidence from German policyDivision of collaborative research grants by type of research consortia

0

50

100

150

200

250

300

350

400

450

1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005

Gra

nted

am

ount

in m

illio

n E

UR

(no

min

al)

.

only firms

only science

firms+science

Source: PROFI database from Germany’s Federal Ministry of Education and Research; own calculations.

Policy design: e.g. collaborative research

• Evidence that spill-over effects are present and that treatment effect of collaborative research is larger than for „individual subsidies“

• Also: spill-over effects larger for collaboration with science– Projects more basic, i.e. more generic in terms of knowledge

creation?– Leading to higher economies of scope?

• Branstetter and Sakakibara (2002), Czarnitzki and Fier (2003), Czarnitzki, Ebersberger, Fier (2007), Czarnitzki (2009).

Policy design: type of R&D

• What type of R&D is actually funded by governments in the business sector?

• Is it mainly basic research?• Or rather applied research and technological

development?• Market failure may be larger for basic research than for

other types– Basic research further away from market– Much higher uncertainty about oucomes and industrial

applications

• Does the agency behave similar to a bank? That is, also prefer less risky projects when making a grant decision?

Policy design: type of R&D

  Total(Strategic) Basic

ResearchMixed projects

Experimental

Development and

Prototyping

Number of

submitted projects3506 1389 829 1288

Grant rate 81% 75% 91% 82%

Grant rate of submitted project proposal by type in FlandersNote: The data were kindly provided by IWT Flanders (own calculations).

• Czarnitzki, Hottenrott, Thorwarth (2010) find indeed that firms suffer more from financial constraints with regard to „Research“ than for „Development“

• Also: firms that receive subsidies for basic research show no sensitivity to financial constraints, but non-subsidized do.

Policy Mix: Subsidies versus R&D tax credits

• How should the government decide on which instrument to use?

• Not so much evidence!– Berube and Mohnen (2009) for Canada: among R&D tax credit

recipients, firms that receive direct subsidies invest additional funds.

– Takalo, Tanayama, Toivanen (2009), structual model on application decision of the firm, grant decision of agency (yes/no and subsidy rate), investment decision of firm

– Model allows simulating absence of subsidies, or „tax credits only“ versus „direct grants only“

– Similar effects of subsidies and tax credits

Distributional effects

• Czarnitzki and Ebersberger (2010) apply a standard treatment effects estimation,

• but then use the results to derive a Lorenz curve of R&D concentration for Germany and Finland

• (Why Germany and Finland? Only direct grants, but no R&D tax credits available)

• R&D concentration is lower in actual situation (policy regime with direct subsidies) than in counterfactual situation (absence of any policy)!

Distributional effects

Lorenz-curve for the distribution of R&D personnel (Finland)

Lorenz-curve for the distribution of R&D personnel (Germany)

Going beyond „treatment on the treated“ on innovation INPUT

• It could be of interest whether firms that are currently not benefitting from subsidies would invest more into R&D if they would receive subsidies– „Treatment on the Untreated“

• Some recent evidence: cross-country comparison Spain, Belgium, Germany, Luxembourg, South Africa (Czarnitzki and Lopes Bento, 2010)

• However, also evidence that currently funded companies do not invest more than actually non-funded firms would invest if they received subsidies.

• More research is needed here! Our study suffers from severe data limitations.

• One could also estimate which firms out of non-recipients would invest most if they would get subsidies

– Ongoing research…..

Going beyond „treatment on the treated“ innovation OUTPUT• Does subsidized R&D leads to more innovation output eventually• Subsidized projects may fail more frequently than others. So, more

patents? Higher sales with new products?• After treatment effects estimation, total R&D can be decomposed

into 2 components– The R&D that the firm would have conducted anyway

– and R&D that was induced by subsidy (subsidy + additionally triggered R&D)

• Some evidence that both components of R&D have a positive impact on patents and new product sales (Czarnitzki and Hussinger, 2004, Czarnitzki and Licht, 2006, Hussinger, 2008).

• However: subsidized component‘s productivity is slightly lower then pure privately financed R&D. (consistent with neoclassical theory)

• Limitations: Timing of output relative to input?

Conclusions and discussion

• Many research questions have been addressed.• Yet, the evidence for actual policy making is still limited• Questions:

– Young Innovative Companies (YICs), see Veugelers (2009), Schneider and Veugelers (2010)

– What is the „optimal“ policy in terms of size of tax credit or mix with direct R&D grants?

– What is a superior design of a funding instrument, e.g.:• Application and grant decision versus:• 2-stage process as in U.S. SBIR program: 1st stage conceptual feasibility

study (small amount of subsidy), 2nd stage large subsidy (only 30% of 1st stage participants survive, on average).

– Sound cost-benefit analysis: treatment effects vs. cost for society.