holistic and timely monitoring of sti system through an annual panel survey to experts and...

Holistic and timely monitoring of STI system through an annual panel survey to experts

and researchers — an experience in Japan

20 September, 2016Masatsura IGAMI

National Institute of Science and Technology Policy (NISTEP), Japan

OECD Blue Sky ForumSTI actors: the potential of direct surveys

1

2

• Japan’s science and technology policies are promoted based on a Science and Technology Basic Plan (STBP) that is formulated every five years.

• The fourth STBP (FY2011 to FY2015).– Identification of important issues that must be tackled.

(Green innovation, life innovation, reconstruction and revival from the disaster)

– The integrated promotion of science and technology/innovation policies.

• How Japanese STI has changed as a result of the basic plan? – The effective implementation of the plan-do-check-act cycle in STI

policies requires evidence regarding changes in STI during the fourth STBP.

Background

1. Introduction

• Continuity– Status of STI does not change overnight, thereby necessitating

continuous monitoring in order to identify changes. • Comprehensive understanding– Assessing the national STI system requires a comprehensive point of

view. • Measurability– There are often instances wherein quantitative data documenting

STI’s status are unavailable. • Timeliness– Timely monitoring of the impact of various policies is needed.– Annual budgeting process.

Challenges in monitoring the status of STI

1. Introduction

3

• NISTEP conducted an annual panel survey that targeted Japan’s leading researchers and experts (NISTEP Teiten survey).

• Their views were assessed by a set of questions intended to measure their levels of satisfaction regarding issues related to STI.

• This survey can be considered as a health monitoring system of STI.

Our strategy

1. Introduction

STI system in Japan

Subject of monitoring Sensors: recognition of the country’s leading researchers and experts (i.e., combined wisdom).

University

Public research institution

Private firm

4

• Status of Japanese STI and its situational changes were monitored by 57 questions asked annually, in addition to questions specific to certain years.

Question items

• Respondents answer each question based on a six-point scale ranging from “insufficient” to “sufficient.”

• Previous year’s responses were fed back to respondents.

• A short description indicating why they changed their position.

Composition of the 57 NISTEP Teiten survey questions

2. Overview of NISTEP Teiten surveys conducted during the fourth STBP

5

Two groups of respondents

1) University / Public Research Institution (PRI) group (about 1,000)

2) Innovation overview group (about 500)

• Presidents of universities / PRIs• Researchers who were nominated by a head of department or subordinate

agency of universites / PRIs• Directors of the World Premier International (WPI)

research center• Principal Investigators of the Funding Program for World-Leading Innovative

R&D on Science and Technology (FIRST) 　　

• Executives of private firms who are responsible for R&D• Representatives of startup companies, venture capital• Heads of academic-industrial alliance offices in universities• Persons who are working on S&T at a think tank / mass media• Directors of a hospital


6

Timeline of the survey and response rate

86% Mean of 5 years (FY2011-2015)

About 1,000 Respondents who responded 5 years in a row


Launch of the fourthSTBP

Launch of the fifthSTBP

Feb. – Apr. 2012FY2011

Oct. 2012 – Jan. 2013FY2012

Sept. – Dec. 2013FY2013

Sept. – Dec. 2014FY2014

Sept. – Dec. 2015FY2015

Apr. 2016FY2016

1st survey 2nd survey 3rd survey 4th survey 5th survey

Discussion for planning of the fifth STBP at MEXT (Summer of 2014)

Discussion for planning of the fifth STBP at Council for Science, Technology and Innovation (Winter of 2014)

7

Examples of questions and their results


Indexchange

Indexchange

-0.47(-0.1)

Science-1.06

(-0.46)

-1.57(-0.77)

Engineering-0.53(0.07)

Agriculture -0.22

(-0.08)

G1 -0.97(-0.15)

Health-0.24

(-0.06)

G2 -0.55(-0.2)

G3 -0.44(-0.23)

G4 -0.17(0.07)

Q1-18: Status of the baseline funding for executing research and development at universities andPRIs

University

Fiel

d of

dep

artm

ent

(Uni

vers

ity)Public research

institute

Uni

vers

ity g

roup

3 4 5 6

Index Index

2 2

Enou

gh

Not

-eno

ugh

Enou

gh

Not

-eno

ugh

3Attributes Attributes

4 5 62.7(748) 3.0(108)

4.0(122) 3.1(248)

1.7(83)

2.9(148) 2.5(236)

2.2(237)

2.2(160)

3.7(203)

2.6(713) 2.9(105)

3.8(117) 2.9(236)

1.5(78)

2.6(134) 2.3(221)

2.1(230)

2.1(153)

3.5(196)

2.5(720) 2.6(105)

3.4(117) 2.8(235)

1.4(82)

2.3(134) 2.3(223)

2.0(235)

2.1(153)

3.5(196)

2.4(725) 2.4(101)

3.2(114) 2.5(239)

1.6(75)

2.1(134) 2.3(227)

1.9(227)

2.0(158)

3.4(206)

2.3(689) 2.0(96)

2.4(109) 2.6(218)

1.5(75)

2.0(121) 2.3(219)

1.7(221)

1.7(143)

3.5(204)

Note1: Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1% or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.

Note2: The satisfaction index values are on a scale of zero (insufficient) to 10 (sufficient). Values of 5.5 or greater are considered “unproblematic” ( ), 4.5–5.4 “not generally problematic” ( ), 3.5–4.4 “insufficient” ( ), 2.5–3.4 “generally insufficient” ( ), and less than 2.5 “extremely insufficient” ( ).

4.0(122)3.8(117)

3.4(117)3.2(114)

2.4(109)

Result of 2011 survey

Result of 2015 survey

Index

Respondents

8

Free answers of reasons for opinion changeReasons for increased satisfaction Reasons for decreased satisfaction

Increased funding made available by the president or dean of university

Initiatives by the university’s executive directors.

Changes in circumstances owing to a respondent transferring to a different institution.

Management expenses grants decreased considerably over time, necessitating budget cuts.

Sufficient money for research cannot be procured through facility maintenance funds alone.

Insufficient funding for conducting experiment based research activities.

Increased cost of electricity and consumption taxes have forced researchers to bear an increasing proportion of expenses related to the use of common facilities.

Funds are being depleted to cover just printing and postage costs.


Free answers provide us with contexts which underline changes of index.

9

10

RankQuestion

No. Category Question Index changeIndexvalue2015

1 Q1-19 Usability of research expenses in Grants-in-Aid for Scientific Research(KAKENHI)

0.79(0.13) 5.4

2 Q1-22Fostering and securing of specialized personnel to handle operationsnecessary for the smooth execution of research activities (URAs) (inuniversities and PRIs)

0.35(0.09) 2.4

3 Q3-04Cooperation beyond the boundaries of the natural sciences to addresstechnical issues for resolving important issues

0.34(0.07) 3.6

4 Q3-12Unified public-private efforts to implement Japanese technology andsystems to overseas

0.32(0.04) 2.8

5 Q3-03Concentration on core competence in government-led research anddevelopment

0.30(0.10) 3.9

6 Q3-02Implementation of strategies and national projects to resolve importantissues through cooperation between industry, academia, and government

0.24(0.03) 3.6

Top 6 questions in terms of positive change with respect to satisfaction between 2011–2015

Note1: Darker shades in the “index change” column indicate greater change in satisfaction. The top and bottom figures in each row indicate the degree of change in satisfaction between 2011–2015 and 2014–2015, respectively.


3. Situational changes of Japanese STI system during the fourth STBP

Research personnel

Research environment

Industry-academia-government collaboration

Innovation policyBasic research

11

RankQuestion

No. Category Question Index changeIndexvalue2015

1 Q1-18 Baseline funding for executing research and development at universitiesand PRIs

-0.62(-0.19) 2.3

2 Q1-06 Whether or not individuals with the required capabilities are applying todoctoral programs

-0.57(-0.17) 3.0

3 Q1-24Research facilities and equipment for innovative and advanced R&D; andthe training of high-quality human resources (in universities and PRIs)

-0.49(-0.07) 4.4

4 Q2-22Degree of diversity in Japanese basic research as the source of futureinnovation

-0.43(-0.14) 3.0

5 Q2-23Degree of originality in Japanese basic research as the source of futureinnovation

-0.40(-0.16) 3.0

6 Q2-17Amount of indirect funding related to the government's public appeal-based research funds (competitive research funds, etc.) (in universitiesand PRIs)

-0.36(-0.07) 4.0

Top 6 questions in terms of negative change with respect to satisfaction between 2011–2015


Note1: Darker shades in the “index change” column indicate greater change in satisfaction. The top and bottom figures in each row indicate the degree of change in satisfaction between 2011–2015 and 2014–2015, respectively.


Research personnel

Research environment

Industry-academia-government collaboration

Innovation policyBasic research

-0.9

-0.8

0.2

1.0

2.6

2.9

3.0

3.0

-1.4

-1.0

0.0

-0.6

2.0

2.4

2.3

2.4

-5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0

Innovation overview group University and PRI groupDe

crea

sing

Incr

easin

g

Changes in research activities at universities and PRIs over the past ten years by respondent group

Research for the direct purposes of resolving social issues and creating economic valueResearch to fulfill organizational goals (e.g., regional or social contributions)

Research to achieve integration between different disciplines

Research in pursuit of temporary trends

Research on basic technologies required by industry (material testing, etc.)Research that grasps fragmented knowledge from a comprehensive viewpoint

Ground-breaking research to produce new research fields

Exploratory research to find new research themes

Note: Respondents were asked to indicate the degree of change compared to 2005 by selecting “decreased greatly,” “decreased,” “no change,” “increased,” and “increased greatly.” The results obtained after indexation were as follows: “decreased greatly” (-10 points), “decreased” (-5 points), “no change” (0 points), “increased” (5 points), “increased greatly” (10 points).

12


13

• The results of the NISTEP Teiten survey showed that respondents felt that some progress was made in resolving important issues during the fourth STBP.– Concentration on core competence in government-led R&D– Implementations of strategies and national projects.– Increase of research for the direct purposes of resolving social issues

and creating economic value.

• However, there is growing concern regarding fundamental research activities at universities and PRIs.– Baseline funding in universities / PRIs.– Degree of diversity/originality in Japanese basic research.

Overview of changes in the status of Japanese STI between 2011–2015


• This paper presented a method for qualitatively determining the status of STI through a panel survey to researchers and experts (NISTEP Teiten survey).

• NISTEP Teiten survey is effective in comprehensive monitoring the STI system’s status.

• The data provided by these surveys are useful in formulating science and technology policies, and have been referenced in official documents by various governmental councils and committees.

• Utilizing qualitative data from the NISTEP Teiten surveys along with R&D statistics would promote a fuller understanding of STI systems.

Conclusions

5. Conclusions and future works

14

Other facts of the NISTEP Teiten survey


13 reports in five years

2 workshops

2 infographics

16 newspaper articles

2.5 million characters in FAs (equivalent to 25 paperbacks)

15

• Fifth STBP started from FY2016.• Preparation for a new survey is

undergoing.

• Fifth STBP identified numerical targets/indicators that will be monitored during the plan.

• Original evidence that NISTEP Teiten survey provides is getting more important in order to understand a context of changes of the indicators.

Future works


16