1 helsinki university of technology systems analysis laboratory multicriteria methods for the...

1

Helsinki University of Technology Systems Analysis Laboratory

Multicriteria Methods for the Shaping of Research Multicriteria Methods for the Shaping of Research

Priorities - Priorities -

A Case Study for Finnish Forestry and Forest IndustriesA Case Study for Finnish Forestry and Forest Industries

Ahti Salo and Juuso Liesiö Systems Analysis Laboratory

Helsinki University of Technology P.O. Box 1100, FIN-02015 HUT

Finland


ContextContext

Research and technology programmes

Earlier engagements – mid-term evaluation of national technology programmes in electronics and

telecommunication (Salo and Salmenkaita, 2002)– ex post evaluation of WoodWisdom, cluster programme for forestry and forest

industries (Salo, Gustafsson and Mild, in press)

Case study – preparation of a new research programme for wood materials science

» international through Scandinavian collaborations

– problem context initially characterised by high uncertainties, vague alternatives, inarticulate criteria for the setting of priorities


Research and technology programmes Research and technology programmes

Characteristics – provide additional funding to basic and applied research in areas that are

deemed vital for long-term industrial competitiveness– are the key strategic instrument of Finnish innovation policy – a large number of on-going programmes

» some 40 programmes funded by the National Technology Agency » 15 programmes funded by the Academy of Finland

– usually include a wide range of supporting co-ordination activities » workshops, seminars, communication activitites

Issues in preparation – what specific research topics should be included in the programmes? – what proportion of funding should be allocated to the different topics?– what horizontal measures are needed to support the uptake of results? need for extensive consultation and validation


Wood WisdomWood Wisdom

Background– the Government launched a programme of increased R&D funding in 1996 – a total of 100 MEUR allocated to seven cluster programmes

» influenced by Porter’s work on industrial clusters » promotion of collaboration among ministries, funding agencies and researchers

WoodWisdom– largest of the cluster programmes with a total funding some EUR 33 million

» National Technology Agency (44%); participating companies and organisations (33%), Academy of Finland (15%); Ministry of Agriculture and Forestry (7%); Ministry of Trade and Industry (2%)

– wood as raw material in the pulp and paper as well as wood products industry» 4 research areas (raw materials, mechanical forest industry, chemical forest

industry, and the operating environment of the forest industry)

– 21 thematic areas, 34 research consortia, 156 projects– 53 companies, 67 research units and 789 researchers


Prospective evaluation Prospective evaluation

Approach – define the objects of inquiry

» e.g., projects, research areas, programme-level activities

– develop an appropriate methodological evaluation framework» e.g., multicriteria decision models

– appoint the workshop participants » interest, competence, balance of stakeholders» consult the funding agencies

– make use of advanced ICT tools » solicit viewpoints from all the participants » allow for anonymous feedback as well » synthesize and discuss results “on the spot”

Remarks – complements but does not replace other forms of evaluation research – may be helpful in deriving recommendations


Examination of future research needs Examination of future research needs

A) What objectives should be stressed in this consortium in the future?

Assign 100 points to complementary objectives at each level of the hierarchy

Main goal First-level criteria Second-level criteria Third-level criteria

Basic research

Applied researchS&T research

Product development

Economics and business re-search

Environmental research

Strengthening ofresource inputs

Other research areas

Social research

Extended collaboration withinexisting networksCollaboration among research

organisations (domestic)Creation of new networks

Extended collaboration withinexisting networks

Collaboration between industryand research organisations (do-mestic) Creation of new networks

Long-termindustrialcompetitive-ness

Development ofresearch collabo-ration

International research collaboration


Wood Quality Variations

18.8

17.7

9.7

11.5

1.5

3.0

8.1

6.7

8.1

9.1

5.8

0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0Basic research

Applied research

Product development

Economic research

Environmental research

Societal research

Increased collaboration betweendomestic research organisations

Creation of new collaborative networksbetween domestic researchorganisations

Increased collaboration betweendomestic research organisations andfirms

Creation of new collaborative networksbetween domestic researchorganisations and firms

International research collaboration

Wood quality variations Wood quality variations


Feedback on the workshops

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

The evaluation resultsprovided constructive

feedback and supportedthe discussion

The workshop results wereuseful and well-founded

The workshops benefitedfrom the GSS use

The evaluation wassuccessful as a whole

Similar workshops shouldbe organised in future

programmes

Strongly agree Agree Indifferent Disagree Strongly disagree Don't know No reply

Feedback on the self-evaluation Feedback on the self-evaluation


Context of decision supportContext of decision support

Earlier programmes– Wood Wisdom cluster programme concluded in February 2002

» strong support expressed for continued research in wood materials science

– desire to initiate a collaborative Scandinavian research programme » joint projects with funding from two or more countries » means of securing higher visibility vis-à-vis European funded projects» sector faced with increased competition from developing countries

Challenges– the scope of the new programme was unclear in Spring 2002

» ’messy’ research topics, approach and objectives, no formally approved criteria » integration of national and international aspirations

– need for to a structured consultation process consultation - how to obtain structured inputs into the planning process? validation - how to validate inputs suggested by researchers and industrialists? shaping - how to generate ideas for the shape of suggested approaches?


Systematic process structuring Systematic process structuring

Development of domain model– a taxonomy of research topics

» three research areas - one Finnish workshop for each » sixteen research themes - structured under the research areas » research topics - proposed by the research resaerchjers

– construction of a multi-criteria model for the analysis of research themes

Preparatory consultation– circulate a survey to some 60 respondents – structured around research areas and themes

» respondents requested to (1) specify the topic and (2) to justify it in detail

Participatory workshops– examination of survey results – assessment of research themes with the help of the multi-criteria model– development of suggestions for funding allocations


Wood or fibre based composite structures

New materials from wood-based polymers or extractives

Biotechnical, chemical or physical modification of wood raw material

Innovative applications of traditional wood and fibre products

Methods of controlling market-oriented utilization of wood raw material

Socio-economic aspects treated as a horizontal theme

Five research themes Five research themes


Workshop objectivesWorkshop objectives

Validation of preparatory work – do the results of the preparatory work fully reflect future research needs? – what further topics should be pursued in the future?– how such this collaboration be managed ?

Shaping of research priorities – how do the research themes relate to the given criteria?– what research themes should the most funding be given?– what considerations should be accounted for in the shaping of priorities?

» e.g., dedicated infrastructures

International research collaboration– which research themes call for collaboration beyond Scandinavia? – with whom should internation collaboration be launched?

» EU (Framework Programme VI), US, Japan, Far East …

– what measures should be taken to promote such collaboration?


Distributed decision support system Distributed decision support system

Client-server architecture– tailored on the basis of the RICH Decisions-software (http://www.rich.hut.fi/)– client-nodes

» solication of evaluations and written comments from the experts

– server » aggregation of evaluative statements and written comments » presentation of results using several modes

– linked via a wireless local area network

Weighting of criteria– ’correct’ criteria weights difficult to obtain

» depends in part on the research theme in consideration» Rank Inclusion in Criteria Hierarchies (Salo and Punkka, 2003)

– ”novelty of topics and research competencies equally important”– ”industrial relevance and capabilties for exploitation equally important”


Agenda Agenda

Introduction 30 min– presentation of participants, workshop agenda, objectives and tools – results from the preparatory work in Finland and Sweden

Analysis of research themes (à 30 min) 2 h 30 min – initial comments on presentations (10 min) – appraisal of research themes (10 min)– discussion (10 min)

Identification of focal research topics 30 min

– results from the appraisal of research themes (10 min)– proposals for resources allocation (10 min)– discussion (10 min)

International collaboration 30 min


Considerations Considerations

Time horizons – roughly about 5-10 years - subject to discussion – what should be achieved through the programme in view of the longer term?

Constructive appraisal of research themes – what measures should be taken to foster innovative capabilities within each of

the research themes? – what specific research topics would be particularly promising?

International dimensions – what priorities should be set for Scandinavian research collaboration – what kind of research collaboration is needed beyond Scandinavia?


General assesment

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

The w orkshop as a w hole w assuccessful.

The results of the w orkshop w ereuseful.

Similar kinds of w orkshops shouldbe organized in the context of otherreseach and technology programs.

1 = Fully disagree

2 =

3 =

4 = Indifferent

5 =

6 =

7 = Fully agree


Observations (1/2)Observations (1/2)

Combinationof formal analysis and informal discussions – a joint understanding ’units of analysis’ obtained after discussions only

» survey and discussions

– formal analysis helps in the pooling of aggragate results – positioning of themes does highlight differing viewpoints

» still a rather subjective presentation» complemented by informal discussions

Structure of the multi-criteria model – relatively small models may work best

» more time to explain what the criteria are intended to mean

– at times rather large models have been suggested » not all criteria are applicable to all research themes » only a limited amount of time can be devoted to each theme» less time to generate qualitative insights


Observations (2/2)Observations (2/2)

Process redundancies – multiplicity of approaches may contribute to validty – direct questions vs indirect analysis

» if allocation of resources is a key question - pose it directly as such! » indirect analysis (multi-criteria) may justify conclusions

– exploration of differences » these may reveal hidden assumptions that are not otherwise accounted for

Decision support system– allows for systematic elicitation of inputs from all participants

» level of expertise also accounted for in the Finnish workshops » anonymous comments also accommodated

– a written track-record produced for later dissemination» less need to write consultancy reports


Publications (2001Publications (2001))Gustafsson, J., A. Salo, T. Gustafsson (2001). PRIME Decisions: An Interactive Tool for Value Tree Analysis, in: M.

Köksalan, S. Zionts (eds.), Multiple Criteria Decision Making in the New Millennium, Lecture Notes in Economics and Mathematical Systems 507, Springer-Verlag, Berlin, 2001.

Salmenkaita, J.-P, A. Salo (2002): Rationales for Government Intervention in the Commercialization of New Technologies, Technology Analysis & Strategic Management 14/2, 183-200.

Salo, A. (2001): Incentives in Technology Foresight, International Journal of Technology Management 21/7-8, 694-710.

Salo, A., K. Cuhls (2003): Technology Foresight - Past and Future, Journal of Forecasting 22/2-3, 79-82.

Salo, A., T. Gustafsson (in press): A Group Support System for Foresight Processes, International Journal of Technology Management.

Salo, A., T. Gustafsson, R. Ramanathan (2002): Multicriteria Support for Foresight Processes. Journal of Forecasting 22/2-3, 235-256.

Salo, A., R.P. Hämäläinen (2001): Preference Ratios in Multiattribute Evaluation (PRIME) – Elicitation and Decision Procedures under Incomplete Information, IEEE Transactions on Systems, Man, and Cybernetics 31/6, 533-545.

Salo, A., O. Kuusi (2001): Developments in Parliamentary Technology Assessment in Finland, Science and Public Policy 28/6, 453-464.

Salo, A., T. Käkölä (in press): Groupware Support for Requirements Management in New Product Development, Journal of Organizational Computing and Electronic Commerce.

Salo, A., T. Gustafsson, P. Mild (in press): Prospective Evaluation of a Cluster Program for Finnish Forestry and Forest Industries, International Transactions on Operations Research.

Salo, A., Salmenkaita, J.-P. (2002): Embedded Foresight into RTD programs, International Journal of Technology, Policy and Management 2/2, 167-193.

1 helsinki university of technology systems analysis laboratory multicriteria methods for the...

Documents