New Research Evaluation

Model: Indentifying Obstacles and Remedies

Jerald Hage, Director Center for Innovation, University of MarylandCollege Park, Maryland

New Ways of Thinking about Research Evaluations

Five Ideas:1.Emphasize the evolution of science and technology or knowledge generally;2. Connect the outputs to larger societal

objectives, ideally comparatively;3. Recognize multiple levels of analysis -—at least three if not five;4. Focus on obstacles and potential remedies at each level; 5. Remedies come from social science theory.

Larger Objective: Decline in US Economic Growth

Five Signs of Decline:1. Negative trade balances in most high

tech sectors, niches;2. Lower rate of radical innovations of

many high tech companies;3. Decline in patents of some high tech

companies;4. Weakening in wage levels of educated

workers;5. Loss of employment in many high tech

sectors.

Comparative Data: Research

Nation RDT% Patents/ GDP Million People

Israel 4.15% 49

Sweden 3.75 100

Switzer. 3.00 115

South Korea 3.21 40

Japan 3.44 107

Germany 2.53 72

U.S.A. 2.77 46

Economist 1 October, 2010: 97

Evolution of Scientific and Technological

Research: U.S.Growth in Knowledge leads to:

• differentiation of research organizations in arenas

• evolution of new networks to connect arenas

Public research laboratories

Pharmaceutical companies

Universities

Bio-tech companiesManufacturing researchQuality researchCommercialization research

Basic research

Applied researchDevelopment research

The Role of Theory in Policy Evaluation of Research

Advantages:The evolutionary theory of knowledge growth

allows for predictions to be made;Patterns of failed evolution are specified and

different in U.S. and Europe The causes lie at multiple levels; Search for obstacles and remedies in social

science theory and beyond in other countries

Obstacles Preventing Economic Growth: Research

Obstacles: Managerial and Policy Remedies:oStatic strategies Seize evolutionary opportunities oLow risk research Broaden research team vision oLack of learning Stimulate cross-fertilization of ideasoStove pipes Integrate the organizationoReactive leaders Appoint transformational teamsoValley of death Construct network coordination oNo industrial policy Public/private cooperation oNon-visible Blockages Perform timely feedback on

innovation at team level and blockages

Behind each obstacle are multiple blockages and the remedies have alternatives, each with problems

Obstacle: Static StrategiesEvolutionary changes:Growth in the number of radical innovations; Globalization in the development of radical

innovations;Differentiation in consumer tastes;Changes in the way in which radical

innovations are produced; Expansion in the number of non-economic

concerns;Emergence of a new stage of manufacturing.

Remedy: Seize Evolutionary Opportunities

Appropriate Response:customize products/services;recognize the evolution in the patterns of doing research;solve national concerns and global problems;increase flexibility in manufacturing by being able to change the composition of parts.

Obstacle: Low Risk ResearchEvolutionary changes in research problemsThe cost and risk (meaning choice of wrong

solution) have increased;The variety of arenas that have to develop

radical solutions for radical product innovation are now greater;

More need to eliminate multiple kinds of risks to the environment and to the individual;

Global solutions have to be customized for specific regional and national differences.

Remedy: Broaden the Vision

Mechanisms for Increased Diversity; Increase the emphasis on radicaliness of

innovation in each of the arenas; Add new kinds of specialists to team; Change problem so that it considers

processes and multiple levels; Explore more arenas of research; Create redundancy between differentiated

research arenas to facilitate the transfer of tacit knowledge

Remedy: Cross-fertilization of Ideas

Mechanisms for More Communication: information that reduces the distance

somewhat between individuals; emotional bonding that slows the rate of

decline in effective communication as diversity increase;

continuously learning that raises the equilibrium curve between effective communication and radical innovation

Obstacle: Stove Pipes

Evidence forStudy of universities

IBM history

GM history

SEMATECH history

Remedy: Integrate the Organization

Mechanisms of integration: Complex organizational strategy that is

visionary

Transformational leadership team

Multiple sources for funding

Obstacle: Reactive Leaders

Evidence for:Failure to adopt flexible manufacturing

Decline in innovativeness in large firms

Loss of manufacturing jobs

Loss of technologies developed in the US

Remedy: Transformational Leadership Teams

Characteristics of: Overcome obstacles; Expand the frontiers by innovations in

occupational specialties and research technologies;

Change the “know-how” of relevant researchers and professionals;

Work on a global problem.

Obstacle: Valley of Death

Evidence for:Study of Brookhaven

Trade deficit in health care products

Lack of public-private cooperation in manufacturing

Remedy: Construct Networks

Modal Types: Japan’s supply chains

SEMATECH as a research consortium

Silicon Valley as a knowledge pool

Obstacle: Blockages in Organizations

Kinds of Blockages:

1. Absence of team processes associated with innovation;

2. Lack of certain kinds of communication;

3. Inappropriate managerial style;

4. Inadequate reward structure for researchers;

5. Not enough research resources;

6. Absence of long term strategy.

Jordan’s Environmental Research Survey

Remedy: Feedback

Levels of Feedback

1.Connecting measures of technical progress to organizational blockages;

2.Situating the organizational research outputs relative to the sector nationally and internationally;

3.Counting the differentiation of research organizations;

4.Measuring the degree of integration between them.

Implications for Data Collection by NSF, STPI, IDA, USDA, etc.

1. Funding and capabilities in each of the six arenas in each sector/niche;

2. More measures of radical innovation performance in each arena;

3. Specialized studies of evolution and failed evolution in a specific sector/niche;

4. Specialized studies of the causes of failure to have radical innovation in each arena of a sector.

Measurement is necessary to pinpoint both failure and its causes

Summary

The Proposed Evaluation Model:

1.Uses a model of knowledge growth;

2.Connects specific research projects to larger issues;

3.Operates at multiple levels;

4. Identifies obstacles and suggests remedies;

5.Enriches theory and vice-versa