follow-up evaluation to advance r&d policy - scenario … · history of evaluation in ......

Follow-up Evaluation to Advance R&D Policy

- Scenario from long term vision -

Yasukuni OKUBO, [email protected]

Director for Technology Evaluation,Technology Evaluation and Research Division

Ministry of Economy, Trade and Industry

Joint TIR/RIHR Workshop on Enhancing Research Performance through Evaluation and Priority

Sept. 15-16, 2008

Contents

Introduction of evaluation at METI and others

2

Introduction of Follow-up Evaluation at METI –Lithium Battery-Introduction of Follow-up Evaluation at NEDOConclusions

01Jan Restructuring of Central Government 22 Ministries & Agencies -> Cabinet Office+12 Ministries

96 1st Science and Technology Basic Plan97 General Guideline for Evaluation of Research and Development

History of evaluation inJapanese public institutions


01Mar 2nd Science and Technology Basic Plan01Apr Independent Administrative Institutions (IAI)

to separate Operational Sectors from Central Government01Jun Government Policy Evaluation Act01Nov Revision of General Guideline for Evaluation of Research and Development02Jan Establishment of The Council for Science and Technology Policy (CSTP)02Apr METI Guideline for Technology Evaluation02Apr METI Standard Items and Criteria of Evaluation

05Mar Revision of General Guideline for Evaluation of Research and Development06Mar 3rd Science and Technology Basic Plan

Grant & contract ResearchCreation of technology

seedsEstablishment of new technology for next

generation

Tax Reduction

Tax credit on increased

R&D expensesNEDO

METI (17% of total S&T budget)MEXT(64%)

METI’s Role in S&T AdministrationMETI’s Role in S&T Administration

N ti l P j t

Subsidy etc.

503 BillionJY

217BillionJY

Subsidy etc.


4

AISTNational Institute of

Advanced Industrial S&T

Industries

Reform of national innovation system(industry - university cooperation, national project reform, IPRs, etc,)

UniversitiesCooperation

Economy

TechnologyTransfer

National Projects(space, nuclear)

TLO

60BillionJYNew Energy and Industrial Technology Development Organization

Mission: Securing sustainable economic growth and stable growth of international economy, through enhancement of Japanese competitiveness, creation of new market,

and improvement of socioeconomic infrastructure.

Policy and program system at METI

6 policies/34 programs /Over 120 projects

1. Economic and Industrial Policy2. Foreign Economic Policy3. Policy for Manufacturing I d t i I f ti &

Programs Projects

Programs Projects

P


6. Nuclear and Industrial Safety Policy

Industries, Information & Communication, and Services4. Policy for Small and Medium-sized Enterprises & Regional Economy and Industry5. Energy & Environmental Policy

Programs Projects

Programs Projects

Programs Projects

Programs Projects

METI Time Table and Methodology０th Year １st Year ・・・・・・ Next Year After 5 Years

Ex-ante Interim Ex-post Follow-up

Start End

Sub-committee

Panel meeting WG

SelfExternalM

onito

ring

Mon

itorin

g

Sub-committee

Panel meeting

ExternalPanel meeting

External


Hearing ● ● ●

Logic Model ○ ○ ○ ○

Indicator ● ● ● ● ● ●

Score ● ●

Peer Review ● ● ●

Expert Review ● ● ●

Quantitative（Bibliometrics）

○ ○ ●

Interview/Questionnare

○ ● ●

Range of Evaluation at R & D Projects at METI

研究開発プロジェクト

インプット資金

アクション

研究

カス

直接

ア

プロ

ジ施策成果

プロジェクトの範囲施策の範囲

アウトカム

インパクト



アクション

研究

カス

直接

ア

プロ

ジ施成果


アウトカム

インパクト１２３



アクション

研究

カス

直接

ア

プロ

ジ施策成果


アウトカム

インパクト

R & D Project

mer

Range of ProjectRange of Policy

Outcome

Impact

of P

olic

y

urpo

se

tcom

e

put M

easu

res)

nage

men

t)

tion

Org

aniz

atio

n)

Evaluation of Impacts to industry and Cost Benefits

Outputs Evaluation


7

資金期間組織

研究テーマ

事業テーマ

スタ

マー

アウ

トカ

ム

ジェ

クト

目標

策目

標(成果指標）

ここまではしっかり描くこの部分は簡単に描いてみる

資金期間組織

研究テーマ

事業テーマ

スタ

マー

アウ

トカ

ム

ジェ

クト

目標

策目

標(成果指標）


４５６

資金期間組織

研究テーマ

事業テーマ

スタ

マー

アウ

トカ

ム

ジェ

クト

目標

策目

標(成果指標）


Cus

tom

Proposal for better strategy

Obj

ectiv

es

Proj

ect P

u

Dire

ct O

ut

Out

puts

(Out

p

Actio

n (R

&D M

a

Inpu

ts (S

alar

y D

urat

In the innovation policy, outcomes are significant.It is not easy to evaluate the outcomes, butWe have been challenging to evaluate outcomes in project evaluation.

Range of METI, AIST & NEDO in terms of R&D



アクション

研究

カス

直接

アウ

プロ

ジェ

施策成果


アウトカム

インパクト



アクション

研究

カス

直接

アウ

プロ

ジェ

施策成果


アウトカム

インパクト１２３



アクション

研究

カス

直接

アウ

プロ

ジェ

施策成果


アウトカム

インパクトMETI

Basic Research

ＮＥＤＯ

Applied Research

Executing Body

Regulation, Tax, etc.

Purp

ose

Range of Policy

ImpactOutcome

es o

f Pol

icy


Direct Outcome

Project Manager

8

資金期間組織

研究テーマ

事業テーマ

タマ

ー

ウト

カム

ェク

ト目

標

策目

標(成果指標）


資金期間組織

研究テーマ

事業テーマ

タマ

ー

ウト

カム

ェク

ト目

標

策目

標(成果指標）


４５６

資金期間組織

研究テーマ

事業テーマ

タマ

ー

ウト

カム

ェク

ト目

標

策目

標(成果指標）


AIST

（Executing Body）Customer

Responsible Range of National R&DPr

ojec

t P

Range of Project

Obj

ectiv

e

NEDO: New Energy and Industrial Technology Development Organization

AIST: National Institute of Advanced Industrial Science and Technology

TLO, etc

Introduction of Follow-up Evaluation at METI

NEDO’s Follow-up and METI’s Follow-up

METI

Executing Body

pose

Impact

fPol

icy


Range of METI’s Follow-upRegulation, Tax, etc.

OutcomeRange of NEDO’s Follow-up

pose

9

ＮＥＤＯ

AIST

（Executing Body）TLO, etc

Proj

ect P

urp

Range of Policy

Obj

ectiv

es o

f

CustomerRange of Project

Proj

ect P

urp

Evaluation Items for Follow-up Evaluation at METI

Ⅰ．ImpactsⅠ－１． Technology ImpactⅠ－２． R&D Performance ImprovementⅠ－３． Economic BenefitⅠ－４ Social Life Improvement

Extent of progress


E id

10

Ⅰ ４． Social Life ImprovementⅠ－５． Feedback to Policy

Ⅱ．Evaluation from current positionⅡ－１． Justification of projectⅡ－２． TargetⅡ－３． PlanningⅡ－４． Justification of ex-post evaluation based on results of Ⅱ－１～Ⅱ－３Ⅱ－５． Follow-up after project

Accelerating or Impeding Elements to Outcomes

Evidences

Role of Follow-up EvaluationIntroduction of Follow-up Evaluation at METI

Few evidences remain

Missing linkages for negative elements

11

295,624

248,207250,966

305,548

293,751289,148

30,00060,000

400,000

600,000（百万円）

自動車用

小型民生用将来的市場ポテンシャル大

30万台/年として2020年600億円

HEV 15万台/年として2010年300億

リチウム二次電池市場将来予測Perspective of Lithium Battery MarketMillion Yen

High Potential in Market

0.3 M cars/ya -> 60 B.Yen in 2020

0.15 M cars/ya -> 30 B.Yen in 2015

HEV

CarsSmall-scale Private Use

Strong point:Higher energy density storage than Lead-acid battery.

Introduction of Follow-up Evaluation at METI–Lithium Battery-

12High Performance Battery Project

Lithium Stationary Battery Project200720062002

2001

300,000 300,000

0

200,000

2000 2001 2002 2003 2004 2005 2010 2020 （年）

小型民生用は3000億円/年レベルで推移300 B.Yen for Small-scale Private Use

(Year)

2011Lithium Battery for FC Project

1992 2006 2010Decentralized Battery Project

①Hybrid Car (Nissan 2000) Shin-Kobe Electric Machinery made Battery

Application to Railway Car

Application to Large Scale cars

Application to Bike

○Down-sizing and weight

⑩Li Battery mounted Cart (shin-Kobe Electric Machinery 2004) Shin-Kobe Electric Machinery made Battery

Application to Care Goods of Wheelchair, etc.○D i i

Application to Gulf Cart○High Density Output○Module Control

⑪Li Battery Test Module for Satellite (SERVIS-1 Satellite 2003) Mitsubishi Electric ・GS Yuasa Cooperation

Application to Satellite

○Thermal Simulation○Down-sizing and weight reduction by Improvement of energy density

③EV(Nissan 2000) Shin-

②Addling Stop Car (Toyota 2005) Mitsubishi Chemical Made Battery material

○High Density Output○Module Control○Design for Circuit○Material Technology

HEV,PEV etc. for low cost & less pollution


Lithium Battery for Hybrid

⑨LI Battery mounted Wheelchair (Yamaha 2005) Hitachi Vehicle Battery

Car

④Hybrid Bus(MitsubishiFuso Truck Bus 2004) GS Yuasa made Battery

⑧Li Battery mounted Diesel Car

(East Japan Railway Company 2007 Test Car) Shin-Kobe Electric Machinery made Battery

⑦Motor Scooter(Yamaha 2003) Hitachi Vehicle Battery

⑤Hybrid Truck (Isuzu 2005) Hitachi Vehicle Battery

⑥Motor Assist Bike(Yamaha Sanyo made Battery

○High Density Output

○Large-scale cell ○Large-scale module

and weight reduction by Improvement of energy density

y y○Down-sizing and weight reduction by Improvement of energy density

2000) Shinkobe Electric Machinery made Battery

Output Measures

1MW class safe-first battery storage

Energy density: 70Wh/kg

Lithium Stationary Battery Project 2006 - 2010

Targets :

40000 yen/kWh combined with the wind power generation.


Decline of demand for electricity, cost down, and conventional regulation impeded to create new demand of battery storage system for family.

Life time： 10 yearsTotal system efficiency： >80％Cost: 40000 yen/kWh

gStationary type has a potential to grow for stabilizing power from wind power generation.

Introduction of Follow-up Evaluation at NEDO

◆To ensure accountability to Japanese taxpayers～by monitoring post-project activities of project participants with regard to the

practical application of R&D achievements.

Purposes of Follow-up monitoring at NEDO


15

Evaluation at NEDO◆ To provide feedback for improving NEDO’s R&D

management～by reviewing past post-project evaluations and post-project activities.

◆ To contribute to the planning of future R&D strategies～by interpreting technology trends in post-project research activities.

Prospect for practical use

What is a Follow-up Chart?A follow-up chart is a tool which chronologically identifies significant events in project management.The main objective is to show up “commercialization” or “discontinuation” scenario in an executing body. Monitoring projects every year, NEDO draws the chart based on interviews to a company concerned. An interviewee is a person who understands the whole project and its evaluation. The horizontal axis represents time while the vertical axis represents prospect for practical use of technology. The coordinates along the vertical axis are based on the opinions of interviewees.

Follow up Chart


Prospect for practical use

Time (years)

Planning Project Post-project

Turning points and their influences/effects on management for practical use

Large

Mid

Small Present

Follow-up Chart

Type A Type B

Achieved Commercialization

Until now, NEDO conducted interviews to 75 companies. Out of them, 46 companies achieved commercialization and 23 stopped R&D to achieve commercialization.The analysis of the interviews categorized them into four different types.

Too optimistic to solve technical problems

Four Types of Follow-up Charts

al u

se

use

Accomplish delineation of technical problems at time of planning.


３３／４６１３／４６

technical problems.R&D breakthrough during latter half of project.

Large

Pros

pect

for p

ract

ica

Mid

Small

Large

Mid

Small

Project TimeProject Time

Pros

pect

for p

ract

ical

Steadily overcome technical problems.R&D based on clear vision for social/ user needs. .

Discontinued R&D after Project

Type DType C

Too optimistic to solve technical problems.No R&D breakthrough during latter half of project

Steadily overcome technical problems.No needs and no cost-competitiveness just after

Four Types of Follow-up Charts Introduction of Follow-up Evaluation at NEDO

８／２３１５／２３

during latter half of project.competitiveness just after the project..

Large

Mid

Small

Large

Mid

Small

Project TimeProject Time

Pros

pect

for p

ract

ical

use

Pros

pect

for p

ract

ical

use

Advantages of Follow-up Monitoring and Evaluation

（２）Dissemination and promotionConducting the follow-up monitoring, NEDO sometimes finds that technology developed in the death-valley. If necessary, NEDO would

id fi i l i i d b l

（１） Share knowledgeThe management lessons obtained from follow-up monitoring are accumulated in the “NEDO R&D Project Management Guideline” accessible through the NEDO’s intranet system.


provide financial assistance in order to remove obstacles.

（３）Policy proposals to the governmentNEDO submits policy proposals including results from follow-up monitoring. Reform of laws and the national innovation system is an example.（4）Opinion exchange with industry

The lessons derived from follow-up monitoring results are shared with the companies participating in NEDO projects. Discussions related to policies required to promote innovation are also carried out.

ConclusionsLegitimacy of policy interventions and relevance of evaluations

To monitor diffusion of products Role of ex-post evaluations

To advance strategy on R&D

20

gyBetter link evaluation results with research budgeting mechanisms and decisions

To identify undiscovered R&D program, funding, regulation, etc.

“Policy mix” for research and innovation policiesTo overview and monitor the whole from outputs to social effects

follow-up evaluation to advance r&d policy - scenario … · history of evaluation in ......

Documents