New Direction of National R&D Evaluation System in

Korea2011. 11. 3

Changwhan MaDirector of Performance Policy Division

EVALUATION 2011 @ Anaheim, CA, U.S.A.

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

Contents 13 slides

1

3

5

13

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

1

3

5

13

R&D Program Management Process

R&D Bud-get Survey/Analy-

sisEvaluation

Implementation(Programs/Projects) In-depth

RecommendationFeedback

Evaluation strategy & Data collection

Self → Meta

Evaluation processUtilization process

Feasibility Study (Ex-ante)

Program (Plan-ning)

Policy (Planning)

1

National R&D Evaluation System in Korea 2

(Guidelines)

Self Eval.

Results (Grades, Recommendations)

Meta EvalProgramEvaluation

GRIEvaluation

Ex-post

In-depth Eval

Ex-post/post Follow-up Survey Follow-up Eval

Management& Research Self Eval. Meta Eval

Ministries NSTC/(KISTEP)

Feedback

Report to NSTC

Efficiency/Effectiveness Of R&D Investment ↑

Preliminary Study

FeasibilityStudy

FeasibilityStudy

Results (Budget Size,Recommendation)Ex-ante Program

Planning

Ex-post

NSTC/MOSF/(KISTEP)

*NSTC: National Science & Technology Commission

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

1

3

5

13

Background of New Direction 3

Continuous increase of R&D investment

3rd on R&D investment / GDP (3.74%, 2010), 14.89 billion USD in 2011

Quantitative Efficiency of R&D Performance has been continuously increased

(Publication and Patent) Near top 10 in the world

Qualitative Excellence is stalled around 30th in the world

S&T and Socioeconomic Competitiveness depend on

the qualitative excellence of researchers and their performance

Therefore, evaluation system should meet the direction of S&T advancement

And new NSTC has recently been launched to respond to the recent change.

New NSTC 4

8

ce

삭제

Ministries, GRIs etc.Planning

ResourceAllocation

Evaluation

NSTC

MoSF

Suggestion about direction of Budget allocation

Allocation · Coordi-nation · Formation

Self evaluation

Meta & Specific evaluation

AfterBefore

Constructing overall R&D System

MoSF Budget Forma-tion

Self evalua-tion

Min-istries

Min-istries

MoSF NSTC

NSTC

Meta & Specific evaluation

Suggestion about direction of budget alloca-tion & Budget allocation / coordination of main National R&D program

NSTC (Multi-departmental R&D pro-gram, etc.)

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

Contents

1

3

5

13

1. Open Evaluation System

10

① More Openness in Performance Information & Expert Participation

To ensure variety of evaluation perspectives by increasing Openness of Performance Information and Various Opinions to the informationTo promote expert review activity by establishing compre-hensive expert community poolTo introduce on specific evaluation first and escalated to other evaluation types

② Establishing Online Open Evaluation System

To share and communicate more by establishing user-friendly online open evaluation system (OOES)To develop model and manual for OOES

5

2. Mission-oriented GRI Evaluation System

11

① Introducing Absolute Rating System

To introduce absolute rating system considering characteris-tics of each GRI (government-supported research institutes)To expand the autonomy of GRI during evaluation processTo focus on accomplishment of GRI missionTo encourage to set challengeable objective to reach world class technology level

② Alleviating Burden on Evaluation

To reduce the portion of management evaluation gradually and integrated into performance evaluation

(depends on revision of act on R&D performance evaluation)

6

3. Rationality and Autonomy of Self Evaluation

12

① Stringent Pre-check of Performance Plan

To ensure the quality of self evaluation by pre-check the ap-propriateness of performance objective and indicatorTo operate official committee on reviewing performance indica-tor

② More Autonomy on Self Evaluation

To set up the evaluation schedule (or cycle) according to milestones of each programs by budget-spending ministryTo carry out meta evaluation in general and re-evaluation on specific programs

7

4. Evaluation of Qualitative Excellence

13

① Qualitative Evaluation considering Impact

To carry out comprehensive evaluation considering scientific/technological and socioeconomic impact and sustainabilityTo encourage to set challengeable performance objective by using qualitative performance indicator

② Guideline on R&D Performance Evaluation

To develop and disseminate R&D performance evaluation guideline for better understanding of qualitative evaluation and for strengthening evaluation capacity of ministries

8

5. R&D Policy Evaluation and Cross-cutting Re-view

14

① Evaluation of R&D Policy and Issue

To introduce R&D policy evaluation for better coordination of R&D programs and R&D environment

② Cross-cutting Review

To expand the scope of specific evaluation from individual program to fields of technology and similar program groupsTo ensure optimize and coordinate the delivery system the similar, overlapping programs

9

6. Strategic Performance Budgeting & Program Improvement

15

① Strategic Performance Budgeting

To set priority and allocate budget of the similar programs ac-cording to evaluation program selection and resultsTo carry out integrated performance budgeting by reviewing mid/long-term direction of R&D investmentTo keep maintaining performance budgeting with the evalua-tion results

② Monitoring System of Performance Information

To establish the DB for evaluation history of the program in-cluding evaluation results, recommendation and management action plan, etc.To ensure the continuous monitoring evaluation feedback ef -ficiently 10

7. Customized Evaluation

16

① Evaluation according to Program Type

To apply differentiated evaluation perspective according to types, size of the programTo develop evaluation model and checklists with common or specialized indicators

② Milestone-based Performance Evaluation

To select the program after considering the cycle or time for producing key performance (creative period; 3 yr, 5 yr, ex-empted, etc.)To encourage to carry out objective and comprehensive per-formance analysis for self monitoring

11

8. Infrastructure and Network

17

① Infrastructure of R&D Performance Evaluation

To establish DB for major performance and budget information as a program management platformTo serve as an active channel for the exchange of information on evaluation trend and various opinions To carry out R&D performance evaluation capacity building

② Network of R&D Evaluation

To hold the evaluation forum regularly (global R&D evaluation network) To carry out collaborative research on evaluationTo participate the conference on evaluation regularly

12

Contents

1. Introduction

2. Background

3. Initiative Focus

4. Conclusion

1

3

5

13

Future Direction 13

Profes-sional

Evalua-tion

Feasible Policy

Optimal Invest-ment

National R&D Program

Qualitatively Excellent Researchers with Perfor-mance

Strong (S&T → National) Competitiveness

Open system

Customized

Mission

…

Thank you!