State of the art insights in capturing, measuring and reporting firm-level innovation indicators

Machteld Hoskens, Julie Delanote, Koen Debackere and Laura Verheyden

OECD Blue Sky ForumGhent, September 2016

Background

• Oslo Manual provides guidelines for measuring innovation at the firm level

• Throughout its 3 editions (1992, 1997, 2005) most attention went to clarifying and expanding innovation concepts

• Questionnaire design received limited attention• For example: just 2 pages out of 163 discussing questionnaire design

issues in 3rd ed., and only 1 paragraph in this section on issues to pay attention to when running international innovation surveys (§ 455)

Concerns

• A small number of empirical studies done so far suggest innovation measurement as we have been doing it for years, is very vulnerable to questionnaire design effects.

Studies

• OECD (2012) study using innovation survey metadata suggested some innovation survey design features negatively impacted the comparability of results between countries.

Studies (ctd.)

• The Netherlands (CBS, 2012) went from a paper form to a web form design in 2011.

• The overall innovation rate increased from 35% to 48%, i.e., a more than 10% increase.

• “Break in series might partially reflect true increase in some innovating enterprises, but more important for explaining the break was the change in survey format.” (CBS, 2012, p. 177)

Studies (ctd.)

• Norway performed a number of randomized experiments in 2011 (Wilhelmsen, 2012). They manipulated 2 factors.

• Innovation rates were lower when the survey was mandatory than when it was voluntary; o e.g., resp. 38% vs. 45% for product and/or process

innovation, a 7% increase.• Innovation rates were lower in a combined R&D – innovation

survey vs. in a stand-alone innovation surveyo e.g., resp. 24% vs. 36% for product and/or process

innovation, an increase of more than 10%.

Studies (ctd.)

• In Belgium we performed a randomized experiment in 2015. We manipulated questionnaire length.

• The innovation rates increased in the short form compared to the regular long form, and sometimes almost doubled.

FormProduct Process Organisational Marketing

1 Long (paper/web) .17 .23 .25 .222 Short (paper/web) .32 .35 .43 .333 Short (phone nrsp) .37 .33 .29 .27

Difference ( 2 - 1 ) .15 .12 .18 .10Difference ( 3 - 1 ) .21 .10 .04 .05

Innovation Type

Studies (ctd.)

• In a World Bank study (Cirera & Muzi, 2016) a short innovation form was administered first, followed by a long innovation form for a random subset of initial respondents.

• Here too, generally higher innovation rates were obtained with the short form.o e.g., resp. 51% and 39% for product innovation for the short

vs. long form, a difference of more than 10%.

• Potential explanations for the findings in the different studies: effects during cognitive processing of survey questionnaire forms: satisficing, unpacking effect, selection bias, mode effects, …

Conclusions

• Innovation measurement at the firm level in its current form is like attitude measurement, highly susceptible to method effects

• This seriously compromises international comparability, as countries currently differ widely in the methods they use

• E.g., within EU, variation in: o voluntary vs. mandatory status of the survey; o use of short form surveys for non-response adjustments; o response mode: paper format, web format, phone interview,

face-to-face interviewo combined vs. stand-alone innovation survey

Recommendations

• There is a clear need for more systematic scientific experimentation with and validation of potential innovation survey design effects

• As innovation measurement occurs in a multinational, multiregional and multicultural context (3MC surveys), we should gain from lessons learned in other 3MC projects such as ESS, PIRLS, PISA, TIMMS, … (see e.g., Harkness, Braun, Edwards, Johnson, Lyberg, Mohler, Pennell & Smith, 2010)o E.g., translation issues should at least be looked intoo E.g., not just cognitive testing but also pretesting in multiple

countries of questions in harmonized EU innovation form

Recommendations (ctd.)

• E.g., More unpacked (list based) measurement of innovation:

“The Oslo Manual presents a rather marked distinction between functional and other forms of changes to products and processes. Future Oslo revision work should attempt to make more explicit what those criteria are and formulate them in survey environments. Specific questions asking firms to describe the various dimensions of novelty may assist in this process. The experience accumulated from this and other related projects on the analysis of design can help formulate these in a more concrete way.” (Galindo-Rueda & Van Cruysen, 2016, § 41, p. 13)

Recommendations (ctd.)

• Revise section below currently in the Oslo Manual, as evidence showed short and long form innovation surveys do not provide equivalent measurements:

• ”For many small units and units in sectors with little innovation activity, the response burden for a full innovation questionnaire may be quite large relative to their innovation activity. Unit non-response rates may also be higher for these units. In such cases, shorter survey questionnaires that focus on a set of core questions can be useful. Short-form questionnaires can also be used in surveying units that have not reported innovation activity in previous innovation surveys. Conversely, for individual units in the above-mentioned groups (small units or less innovative sectors) which have previously reported substantial innovation activity, full questionnaires may be used.” (OECD, 2005, pp. 124-125, § 456)

Recommendations (ctd.)

• Revise recommendation currently in Eurostat methodological guidelines, as evidence showed short and long form innovation surveys do not provide equivalent measurements:

• ”If non-respondents, as an un-weighted percentage of all relevant enterprises in the sampling frame, exceed 30%, then a simple random sample of at least 10% of the non-respondents (excluding non-relevant enterprises) should be selected. The questions to be included in the non-response survey are specified in Annex 7.” (Eurostat, 2014, p. 5)(Annex 7 specifies 8 core questions, i.e., an abbreviated version of the regular long form)