application of various designs in evaluation of regional programs

Anna Borowczak, Ph.D. candidatesupervised by Prof. Pawel Churski

Regional Analysis Department, Institute of Socio-Economic Geography & Spatial Planning

Poznań, 25th of June 2015

1. Reasons justyfying the topic of disseration.2. Scientific objective and research questions,

time-frame and territorial scope.3. Choice of references.4. Data collection and methods.5. Structure of the content.6. Terminology.7. Theoretical background.8. Model of evaluation for regional operational programs.9. Empirical validation of evaluation model with three independent evaluation

designes.10. Conclusions and discussion.

Part of the research presented in this dissertation was funded by the National Centre of Science

in the framework of a research project no UMO-2011/01/N/HS5/00100managed by the Author of this dissertation.

Agenda.

Introduction

• Evaluation is an important tool for public administration, while it serves thepurpose of getting a market feedback, when public bodies need to assess their non-profitinvestments;

• Evaluation researches play key role in regional development, because theyconstitute substantial part of empirical research;

• Nevertheless, evaluation research practice is neither sufficiently supportedby theoretical work, nor it is commensurately reflected in literature on socio-economic geography;

• This may be a major reason for poor methodological quality of evaluation research,which is backed by the following observation: „evaluation researches are characterized by the lack ofconsistent methodology rooted in literature, limited number of methods, lack of more complex cause and effect analysis,reaching beyond planned indicators towards real changes evoked by the intervention, including changes of relations (...)Evaluation designs are often simply compliant with guides issued by the European Commission, being mostly technicaland imitative [Olejniczak, 2009].

• But the other reason lies in intrinsic methodological challengesof evaluation research, stemming from the characteristic co-operation betweenpublic body and evaluator in the process of shaping the methodology of the eachresearch;

Justification of the topic.

This dissertation seeks to:

• Enhance methodological discussion in the field of evaluation of regionaldevelopment.

• Make an atempt to incorporate evaluation research into regional science.• Help public administration to achieve best results with respect to scientific

knowledge.• Help evaluation professionals to design better evaluation studies.• Improve reliability of evaluation studies in regional development.

Gaps addressed in the dissertation.

Scientific objectives & research questions.

Establishing an evaluation model including the outcomes of empirical validation carried out with three independent evaluation designs

Construction of evaluation model for regional programs

Empirical verification and validation

of evaluation model with three independent

evaluation designs

Determining advantages and limitations of each

evaluation design

Main objective

Theoretical and methodologicalobjective:

Cognitive objective:

Applicable dobjective:

Research questions:

1. What theoretical aspects of regional development should be considered whendrafting terms of reference for evaluation research?

2. What are information needs declared by public bodies with regards to investments in regional development?

3. What factors determine the application of particular evaluation design in evaluation of regional programs?

4. Which evaluation design offers relevant information as seen from both practicaland theoretical angles on regional development?

5. Do the outcomes of evaluation designs differ among each other? To whatdegree are they similar/ contradicting?

6. Do the outcomes allow for reducing the level of insecurity while makingdecisions on reshaping the next interventions in regional develoment?

Scientific objectives & research questions.

Subject of evaluation research:embraces investments from two generations of programs: Integrated Regional Operational Program (2004-2006) Regional Operational Program (2007-2013)

Time scope:years 2003 – 2010enables an analysis of two crtical moments:(2003) before any intervention started and (2010) cummulation of investments from two programsbased on accessible data.

Territorial scope: • Poznański subregion (61) – at NUTS 3 level:• suburban region with 565 thous. inhabitants• meso-scale, enables analysing both macro- (measured

GDP) and microeconomic interdependencies.

Time-frame and territorial scope.

Source: Author’s own elaboration.

Choice of references.Ev

alua

tion

Category Author, editionDefinition, typesand functions

Alkin, M., 2004; Chelimsky, E., 1997; Campbell D.T., 1979; Dahler-Larsen P., 2005; Drobniak A., 2009; Kierzkowski T. 2002; Korporowicz L., 1997; Mathison S., 2004; Mizerek H., 2002; Olejniczak K., 2008 (a,b); Rossi P.H., Lipsey M.W., Freeman H.E., 2004; Sanders J., 1994; Scriven M., 1982, 1992; Shadish,W.R., Jr., Cook T.C., Leviton L., 1991; Stern E., 2004; Stufflebeam D.L., Madaus G.F., Kellaghan T., 2000; Vedung, V., 1997.

Methodology Anderson, A.A. (undated), Morton M.H., 2009; Barkley D.L., 2008; Baslé M., 2006; Bradley J., 2006; Chen H. T., 1990, 2004; Churski P., Borowczak A., 2010; Connell, J.P, Kubisch, A.C., 1998; Cook T.D., Campbell D.T., 1979; Donaldson S.I., 2007, Dutkowski M., 2008; Górniak J., Worek B., Krupnik S., 2007; Guba E., Lincoln Y.S., 1989; Haber A., 2007; Henry G.T, 1998; Jasiński M., Kowalski K., 2007; Komornicki T., Rosik P., Śleszyński P., Pomianowski W., 2010; Leeuw F. L., 1991, 2008; Malik K., 2009, 2011; Malik K., HeffnerK., 2010; McLaughlin J.A., Jordan G.B., 1999; Owen J.M., Rogers P.J., 1999; Patton M.Q, 2002; Pawson R., 2006; Pawson R., Tilley N., 1997, 2004; Potter J., 2009; Pylak K., 2009; Roberts P., 2006; Trochim, W., 1989(a,b); Trzciński R., 2009; Weiss C.H, 1972; Wholey J., 1983. W tym uzupełniająco: Antoszkiewicz J., 1990, Babbie, E., 2004, Bickman, L. 1987; Chojnicki Z., 1999, 2008; Gruszczyński, L.A., 2003; Hammersley M., Atkinson P., 2000; Heffner C.L., 2004; Kuhn T., 1968; Nowak S., 1965; Rogacki H., 2003; Such J., Szcześniak M., 2006; Sztumski J., 1995; Tokarski T., 2011.

Choice of references.Re

gion

alde

velo

pmen

tEv

alua

tion

Category Author, edition

Legal and administrativeaspects of regional policy evaluation in Europe & Poland

Bachtler J., 2000, 2001; Bachtler J., Poverari L., Taylor S., 2000; Bachtler J, McMaster I., 2008; Batterbury S., 2006; Bienias S.,2012; Bober J., 2007; Dutkowski M., 2008; Górniak J., 2007; Haber A., 2007; Malik K., Bedrunka K., 2008; Mazur S., 2007; Mierzejewska L., 2009; Olejniczak K., 2009; Żuber P., Bienias S., 2008

Theoreticalaspects of regionaldevelopment

Barro R., Sala-i-Martin X., 1995; Budd L., Hirmis A.K., 2004; Camagni R., 2002; Chądzyński i in., 2007; Chojnicki Z., 1999, 2008; Chojnicki Z., Czyż T., 1991; 2006; Churski P., 2008, 2009; 2011; Czapiński J., Panek T. , 2009; Czyż T., 2008, 2012; Dijkstra L., Annoni P., Kozovska K., 2011; Domański B., 2004; Domański B., Gwosdz K., 2008; Domański R., 2002, 2008; Giannias D., Liargovas P., Manolas G., 1996; Golejewska A., Gajda D., 2012; Gorzelak G., 2003, 2004, 2009; Gorzelak G., Jałowiecki B. 2001; Gorzelak G., Smętkowski M., 2005; Grosse T.G., 2002; Huggins R., Izushi, H., 2008; KlasikA., 2002; Kliber P., Malaga M., 2003; Kudłacz T., 1999; Markowski T., 2009; Molle W., 2007; Nermend K., 2008; Orłowski W., 2003; Parysek J.J., 2008; Porter M., 1990; Potter J., 2009; Sengupta J., 2011; Stackelberg K., Hahne U., 1998; Stimson R., Stough R., Roberts B., 2003; Storper M., 1997;

Choice of references.

Regi

onal

deve

lopm

ent

Category Author, edition

Contemporaryconditions for regionaldevelopment in Europe and Poland

Ahner D., 2008; Bachtler J., Wren C., 2006; Bachtler J., Raines P., 2012, Bachtler J., YuillD., 2001; Barca F., 2009; Barca. F., McCann P., Rodriguez-Pose A., 2012; Begg I., 2001, 2008; Domański B., 2008, 2012; Faludi A., 2009; Fenge R., Meier V., 2008; Fesus G., Roller E., 2011; Gaffey V., 2010; Grosse T.G., 2008; Kukliński A., 2003; Markowski T., 2009; Rodriguez-Pose A., Fratesi U., 2004; Ross-Larson B., 2008; Steiner M., 2013; Strzelecki Z., 2011, 2013; Swianiewicz P., 2011; Szlachta J, 2011, 2012; Vanthillo T., Verhetsel A., 2012; Wierzbicki A., 2008; Zaucha J., Szlachta J., 2012.

Data collection & methods.

Published secondary data:Eurostat, Polish Central Statistical Office, Polish Central Custums Office, evaluation research databases by European Commission and Polish Ministry of Development.

Unpublished secondary data:• project database of Polish Ministry of Development, Regional Voivodship Office

and Regional Marshall Office (n2004-2006=148, n2007-2013=225);• Terms of Reference regarding ex-post evaluation issued by the Central Evaluation

Unit and 16 Regional Marshall Offices in Poland (n=23).

Primary data collection:• semi structured interviews -CATI- computer assisted telephone interview (n=802);• questionnaire addressed to local development experts (n=6).


Structure of CATI respondents(for purposes of theory-driven and particpatory evaluation designs)

Type of respondent Total no of entities

in relevant areaNo of interviews carried out

in relevant area

ENTERPRISES• IROP Program 2004-2006 50 20• ROP Program 2007-2013 220 85• non-participants 73 000 365NGOs

total 1800 300LOCAL GOVERNANCE

local communitiespoviats

375

293

TOTAL - 802


Type of respondent Name

NGOs by category

Economic • National Chamber of Commerce• Cluster Assotiation

Society and Environment Center for Fostering Eco Development

Regional Governance by responsibilty

IROP 2004-2006Regional OperationalProgram (2007-2013)

Voivodship Office

Marshall Office

Scientific environment

--- Institute of Socio-Economic Geography & Spatial Planning (Adam Mickiewicz University)

Structure of expert panel (for purpose of participatory evaluation)


Statistical-quantitative methods• indictator analysis• absolute beta-convergence• principal component analysis• structural equasion models• multidimensional scalling

Social research methods• textual analysis• heuristic methods –concept mapping• field research

Cartographic methods• ArcGIS

Methods & techniques

Concetptualization

logic models, concept mapping

Observationand analysis

structural equasionmodels,

indicators, multidimensional

scalling

Fig. Breakdown of methods according to the stageof evaluation procedure

Source: Author’s own elaboration

Chapter I. Introduction

1.1. Scientific objective and scope of research1.2. Outline of literature1.3. Data sources and methods

Chapter II. Evaluation as a tool for invesitigating public interventionsin regional development

2.1. Definitions and characteristic features of evaluation2.2. Types and functions2.3. Assessment criteria2.4. Methodology and research designs2.5. Systemic conditions of evaluating regional programs under the framework

of EU cohesion policy in Poland

Structure of the content.

Chapter III. Regional development as the topic of evaluation research

3.1. Regional development in Regional Science. What important lessons can fuelevaluation research?

3.2. Evaluation of regional development in perspective of policymakers. What do we thinkwe need to know?

3.3. Model of evaluation for regional programs

Chapter IV. Regional programs in Poland

4.1. Regional programs in the context of EU cohesion paradigm4.2. Regional programs as planning documents. What is planned, where and why?4.3. Regional programs in Poland

4.3.1 Integrated Regional Operational Program4.3.2 Regional Operational Programs


Chapter V. Testing the evaluation model for regional programs in poznanskisubregion

5.1 Contextual diagnosis of regional development in poznanski subregion5.1.1 Economic cohesion5.1.2 Social cohesion5.1.3 Territorial cohesion5.1.4 Competetiveness5.1.5 Living standards and quality of life

5.2 Empirical verification of the evaluation model 5.2.1 Objective-based evaluation design5.2.2 Theory-driven evaluation design5.2.3 Participatory evaluation design

5.3 Comparative analysis of the outcomes

Chapter VI. Conclusions and discussion


Theoretical backgound

Evaluation in a narrow sense:

a single research process, seeking to determine the merit and value of publicintervention, characterized by high utility of evidence-based conclusions and reliable andobjective methods, or a product of such process.

[Scriven 1990, 1991; Stufflebeam et al..2000; Patton 2002; Alkin 2004, Such & Szcześniak 2006, Górniak 2007, Haber2007, Olejniczak 2008]

Evaluation in a broader sense:

a body of normative research aimed at optimization of public interventions,included in the applicable strands of various scientific disciplines, adequate to thegeneric subject of evaluation research. In case of evaluating regional programs,evaluation may be considered an applicable strand of socio-economic geography,providing a respective scientific rigor of evaluation research is maintained;

[Chen 1990, Scriven 1991, Shadish 1991; Chojnicki 1999; Alkin 2004; Mathison 2004]

Basic terminology.

Basic terminology.

Meta-evaluation

analyis of the quality of evaluation research, carried out with respect to the followingcriteria:

• Reliability: predictable, accurate and replicable procedure of research;• Credibility: significant cause and effect chains;• Generalizablity and transferability: universal conclusions;• Objectivism: neutral attitude towards object of research, well-documented oppinions• Responsiveness: utile conclusions for stakeholders and recipients.

[Weiss 1972, Guba & Lincoln 1989, Chen 1990]

Key factor differentiating designs is the way, in which the effects of the intervention aredetermined and priorititized against each other.

1. Objective-based evaluation• explicit objectives as specified through indicators

or stated in the description of intervention

2. Theory-driven evaluation• reconstruction of the logic behind the intervention

based on scientific theories• reconstruction of political, behavioral and social

logic justifying the intervention

3. Participatory evaluation• effects of the intervention as registered and

weighted in the perception of the stakeholders

Theoretical backgound.Evaluation designs.

Hermeneutic circle

Logic model

Paradigm: universally recognized scientific achievements that, for a time, provide model problems and solutions for a community of researchers

[Kuhn, 1968].

Paradigm of regional development:• universally accepted hierarchy of values,

which are measures in meeting social needs[Chojnicki 2008].

• economic doctrine explaining actual mega-trends and mechanisms specified in recenttheories of regional development

[Kudłacz 1999 Stimson 2003, Churski 2008].

Theoretical backgound.Paradigm of regional development.

Source: Author’s own elaboration.

Model of evaluation for regionalprograms.

Model derived from:

• compilation of theoretical acpects of RegionalScience

based on paradigm and final approach to regionaldevelopment (i.e. considered a deliberate changebetween two conditions of the system)

• analysis of 23 ToR’s for ex-post evaluationsdrafted in the 2004-2010;

including measuring impact of intervention on regionaldevelopment and considering functionality of interventions in terms of their morphology and responsiveness to the actual needs (whethergeographical or social).


Outcomes of empiricalverification

Evaluation in poznanski subregion.Contextual analysis.

Diagnosis of socio-economic development in poznanski subregion serves the purpose of being a point of reference to outcomes of three evaluation designes in a further comparative analysis.

It is of a macroeconomic character and refers to the following processes:

• economic, social and territorial cohesion;• changes of competitiveness;• changes in living standards and quality of life.

observed between 2003 (before intervention) and 2010 (final state of the regional system),

and carried out in three spatial scales, whenever possible: (a) all NUTS 3 units in Europe; (b) all NUTS 3 units in Poland; (c) all NUTS 3 units in Poland with regards to the intraregional discrepancies.

Evaluation in poznanski subregion. Socio-economic diagnosis.

Economic cohesion: aboslute beta convergence

EU NUTS III units(EU 27)

ModelNo of observations 1166Changeability 75%- highp (significant for p <0,05 ) <0,00001

Co-efficient of determination R2 0,7 -0,0464113 (momentum of convergence) 0,16

Values of GDP per head in 2010 areestimated in real prices from 2003 . Thisallows to observe real changes in PPP. HCPI index has been used as a GDP deflator.



Economic cohesion: aboslute beta convergence

Polish NUTS III Intraregionaldiscreapancies

ModelNo of observations 66 66Changeability 10,5% 41,7%- averagep (significant for p <0,05 ) 0,09026 0,0008Determination co-efficient R2 - 0,16 - -0,0402308 (momentum of convergence) - 0,04


Social cohesion: spatial distribution of Gini coefficient measuring the intensityof intraregional income disparties broken down into 7 income groups in eachsubregion.


2005 2010


Territorial cohesion:

descriptive change determined throughoutcomes of following research[Korcelli 2007, Konecka-Szydłowska 2009, Churski et .2009 Sierpieński 2010, Komornicki 2012, Rosik 2008, 2012, ESPON 1.13, ESPON 1.44, Spiekerman & Wengner 2006, 2008]: enocompassing:• settlement patterns,• spatial accessibility.


Source: Komornicki &Rosik, 2013 Source: Spiekerman &Wengner, 2006

Competitiveness: spatial distribution of V1 component (PCA) interpreted as structure and effectiveness of employment accross Polish subregions.


Author’s own ellaboration.

2003 2010

Living standards and quality of life: spatial distribution of S1 component (PCA) interpreted as accessibility to public services and effective land usein subregion accross Polish subregions.


2003 2010Source: Author’s own elaboration

Evaluation in poznanski subregion. Outcomes of contextual diagnosis.

Strenghts of poznanski subregion:- confirmed economic convergence with other EU NUTS III units (higher than

average momentum);- confirmed economic convergence with other subregions in region against the

backdrop of all Polish regions (average momentum); - low but fast growing competitiveness as compared to the EU;- high competitiveness vs. other Polish subregions;- high wages and employment level vs. other Polish subregions;- sptatial accessibiity is low when confronting other EU NUTS III units, but one of

the best in country.

Weaknesses of poznanski subregion:• relatively lower accessibility to public services rooted in violent suburbanization

process;• high income polarization, high concentration of incomes in industry and

contsruction sectors, relatively low structural diversification of incomes.

RESEARCH PROCEDURE:

Conceptualization phase: logic model of intervention based on aggregated indicatorsdescribing products and results of IROP and RPO programs.

Observation and analysis phase: indicatoranalysis, regression analysis

Data: quantitative

Source of data: field work collectingthe data on projects stored by the

implementing institutions

148 projects of IROP with 700 indicators; 225 projects of RPO with 70 indc.

Interpretation of the effect causedby intervention: material effect

Evaluation in poznanski subregion. Objective-based evaluation.



Lp.

Wskaźnik produktu

Okres

Wskaźnik rezultatu

Okres

Nazwa Jedn

.

Wartość

Nazwa

Jedn

.

Wartość

obor

nick

i

pozn

ańsk

i

szam

otul

ski

śred

zki

śrem

ski

RA

ZE

M

2000

-200

6

2007

-201

3

obor

nick

i

pozn

ańsk

i

szam

otul

ski

śred

zki

śrem

ski

RA

ZE

M

2000

-200

7

2007

-201

4

1 Długość wybudowanej drogi wojewódzkiej

km

- 16,02 - - 7,75 23,77 A x Poprawa dostępu do terenów inwestycyjnych

ha

170 142 159 8 150 629 x - 2 Długość wybudowanej drogi powiatowej

km

- - - - - 0 A x Liczba inwestycji na terenach inwestycyjnych

szt - 23 2 - - 25 - x

3 Długość wybudowanej drogi gminnej

km

- 6,43 0,64 1,94 1,52 10,53 A x Oszczędność czasu w przewozach pasażerskich

mln

zł

7,2 8,05 3,9 4,04 8,08 31,27 - x 4 Długość zmodernizowanej drogi

wojewódzkiej km

11,53 25,17 36,36 0,45 43,32 116,83 A x Oszczędność czasu w przewozach towarowych

mln

zł

1,8 17,29 16,6 1,46 20,4 57,55 - x 5 Długość zmodernizowanej drogi powiatowej

km

31,44 3,58 17,17 4,07 3,02 59,28 A x

6 Długość zmodernizowanej drogi gminnej

km

- 11,79 0 6,21 3,36 21,36 A x

7 Długość wybudowanych ciągów pieszo-

rowerowych km

1,88 24,02 6,57 0,45 3,01 35,93 A -

8 Długość zmodernizowanych ciągów pieszo-

rowerowych km

2,62 4,7 1,5 1,56 0,8 11,18 A -

In the course of aggregation 36 product and 17 result indicatorswere identified from an extensive group of 700 IROP’s and 70 RPO’sindicators to represent universal and trackable material effects of bothprograms. Then indicators showing the biggest change versus their basevalue were further examined with use of the regression analysis.


Outcomes of evaluation research:

The strongest impact has been confirmed for the following types of effects:

• number of supported SMEs (=0,66);• number of modernized universities (=0,33)• number of waste facilities (=0,33)• modernized sewage system (=0, 028)• modernized water system (=0,005)

Despite the magnitude of the impact, the greatest physical change vs. base indicatorshas been observed for the infrastructural investments rather than in the investementsin economic realm.

These outcomes are partially compliant with socio-economic diagnosis:• They confirm the actual investments were rightously targeted on existing territorial

deficits, such as accessibility to public (communal) services.• But they do not reveal the actual impact on the scale of the deficits in 2010 (the

final state).

RESEARCH PROCEDURE:

Conceptualization phase: reconstruction of logicmodel behind two regional programs based upon regional development theories;

Observation and analysis phase: structuralmodel equasion (SEM);

Data: quantitative;

Source of data: field work, semi-structured CATI interview (n=802);

Interpretation of the effect caused by intervention: magnitude and vector (+/-) of impact on basic parameters of regionaldevelopment.

Evaluation in poznanski subregion.Theory-driven evaluation.



Tab. Standarized estimants of SEM model displaying impact of EU funds on the level of development of poznanski subregion (AGFI=0,923)

ue1 ue2 ue3 DP(s3) K WK1 WK2 PiWŻDP(s3) -0,033 -0,175 0,028 0 0 0 0 0K 0,461 -0,299 0,296 0,358 0 0 0 0WK1 0,084 -0,055 0,054 0,066 0,183 0 0 0WK2 0,141 -0,092 0,091 0,11 0,307 0 0 0PiWŻ 0,088 -0,092 -0,069 0,779 -0,466 0 0 0Y (GDP) 0,342 -0,223 0,214 0,297 0,714 0 0 0,043w5 0,053 -0,034 0,034 0,041 0,115 0,629 0 0w1 0,021 -0,014 0,014 0,017 0,046 0,254 0 0w4 0,046 -0,03 0,029 0,035 0,099 0,54 0 0w2 0,059 -0,038 0,038 0,046 0,129 0 0,42 0w6 0,049 -0,032 0,031 0,038 0,106 0 0,346 0j6s7 0,061 -0,064 -0,047 0,537 -0,321 0 0 0,69j5s6 0,061 -0,064 -0,048 0,543 -0,324 0 0 0,696j3s5 0,066 -0,069 -0,051 0,585 -0,35 0 0 0,751j2s2 0,068 -0,071 -0,053 0,604 -0,361 0 0 0,775j1s1 0,077 -0,08 -0,06 0,678 -0,405 0 0 0,869



Outcomes of evaluation research:• strong impact of investments in economic realm on competitiveness (while

indirectly influencing wages, employment and SMEs’ investments, observinga weaker influence on favorable investment climate);

• investments in transport infrastructure noting positive influence oncompetitiveness;

• investments in spatial accessibility noting only slight impact on quality of publictransport.

Side-effects:• negative impact of investments in social realm on GDP;• negative impact of investments in economic realm on spatial accessibility

(quality of public transport).

Fully compliant with socio-economic diagnosis:• Intervention targeted at the competitiveness, model confirms the deficits as

diagnosed.

RESEARCH PROCEDURE:

Conceptualization phase: concept mapping

Observation and analysis phase: non-structured interviews, multidimensional scalling, assessment by an expert panel.

Data: qualitative

Source of data: field work, CATI in poznanski subregion (n=802)

Interpretation of the effect caused by intervention: qualitative effectsaccording to the perception of inhabitants

Evaluation in poznanski subregion. Participatory evaluation.



1. Determiningthe effects

• Asking CATI respondents on the effects of regional programs in unstructured way

2. Structuringthe effects • Interpreting and ordering qualitative data

3. Assessing the value of effectsand clustering

• Setting the panel of experts• Expert assessment and clustering

4. Representingthe effects on

map of concepts• Using statistical method to compute the

outcomes of qualitative research

5. Interpreting the map of concepts

• Analysis of outcomes• Labelling clusters of similar effects with

names

Procedure of conceptmapping


NAME OF EFFECT

1. Launching a new company

2. Enhancing productivity of local enterprises

3. Building competitiveness of local enterprises

4. Enhancing employment in local enterprises

5. Green investments in local enterprises

6. Improving quality of education:

7. Improving quality of public health services

8. Enhancing investments of local enterprises

9. Enhancing investments of public authorities

10. Enhancing investments of NGOs

11. Decline of public spendings

12. Improving the image of local territorial unit

13. Building sewage plant and systems ….

xx. ….

26. …. International integration of youth

Outcomes of the following stages of concept mappingprocedure:

1. determination2. structurization3. assessment(clustering)

Assessment criteria:relevance, urgenceranging <0,3>


Fig. Similarity matrix (conceptual) Fig. Multidimensional scalling

Outcomes of following stages of the concept mapping procedure:3. Assessment and clustering of the effects4. Representation of the effects



GROUPS No of EFFECT LABELS FOR GROUPS OF EFFECTS RELEVAMCE URGENCEGROUP 1 1,2,3,4,5,8 INVESTMENTS IN LOCAL FIRMS 2,53 2,23GROUP 2 6,14,26 EDUCATION 2,06 1,56GROUP 3 9,11 MAINATINING INVESTMENTS’ DYNAMICS BY PUBLIC AUTHORITIES 1,00 0,50GROUP 4 12, 17 PROMOTION AND REVITAILIZATION 1,50 1,25GROUP 5 16, 19, 18 LOCAL PUBLIC TRANSPORT AND TECHNICAL INFRASTRUCTURE 2,00 1,44GROUP 6 7, 13, 15, 22 IMPROVING QUALITY OF LIFE 2,00 1,38

GROUP 7 24 25 IMPROVING LIVING CONDITIONS OF MARGINALIZED GROUPS 2,00 1,75Single effects: 10 SUPPORT FOR NGOS 1,00 1,75

20 BROADBAND INFRUSTRUCTURE 2,50 2,0021 EQUIPPING GREEN FIELDS WITH TECHNICAL INFRASTRUCTURE 2,00 2,1723 SOCIAL ACTIVATION 2,33 1,83

Fig.: Labelling the clusters of similar effects with common names

Outcomes of the concept mapping procedure:interpretation of clustering



Outcomes of the evaluation design:

In perception of the stakeholders of regional programs the greatest signifcance for the regional development is attributed to the following effects:

• Investments in local enterprises, • Investements in the broadband internet network, • Social activation, • Education (including modernization of educational infrastructure). • Equipping green fields with technical infrastructure

Side-effects:• Enhancing the enterpreneurs’ skills in planning investments• Social activation of inhabitants

Partial compliance with socio-economic diagnosis: intervention targeted in competitiveness.

Conclusions

Comparative analysis of evaluation designs.

Dimensions of comparative analysis of three independent evaluation designsof regional development programs:

• cognitive dimension: degree, to which outcomes of respective designsrespond to evaluation questions specifying research issues following from operationalized model of evaluation for regional programs;

• methodological dimension: degree to which criteria of meta-evaluation aremet in particular evaluation design.

Cognitive dimension of comparativeanalysis.

Level of analysis Component of the model Analytical criteria

Outcomes

Planned effects

Detailed description of effects Hierarchy of effects according to impact’s

magintude Hierarchy of effects according to impact’s

volume

Side-effects Sort of measured side-effects

Mechanisms Construction of intervention according to

policymakers

Accuracy of impact’s measurement vs. assumed level of spatial analysis

Explanation of mechanism of exerting the impact (i.e. mobilizing developmental factors)

Explanation of origins and reasons of side-effects

Theories of regional development

NeedsNeeds of targeted geographical areas

Adjustment of intervention to geographicalareas

Identification of needs that have not beensatisfied

Needs of targeted social groups

Adjustment of intervention to social groups Identification of needs that have not been

satisfied

Cognitive dimension of comparativeanalysis.

General assumption: the more components of the evaluation model are considered in the evaluation design the better.

• The best result is achieved by the theory-driven evaluation.

Limitations: not too detailed description of effects, low degree of determining the correspondance of effects vs. needs of social groups.

• The average result is achieved by participatory evaluation.

Limitations: low chance to determine the hierarchy of effects in comparison to other designs, does not explain the mechanisms of obtaining the effects.

• The weakest result is achieved by objective-based evaluation.

Limitations: does not identify side-effects, does not explain the mechanisms of intervention, only to a small extent allows for assessing the adjustment of intervention to needs of socialgroups, does not identify the unsatisfied needs.

Methodological dimension of comparativeanalysis.

Criteria of analysis

By evaluation design

Objective-based Theory-driven Participatory

Reliability High Average Low

Credibility Average High Low

Generalizibility High Average Low

Objectivism High Average Average

Responisiveness Low High High


Summary. Conclusions. Discussion.Verification of the model of evaluation regional programs including relevant theoretical andpractical aspects of assessing the impact of regional programs on the process of regionaldevelopment, was based on three independent evaluation designs, the procedures of whichhave been adopted to theoretical and practical conditions of measuring the impact on regionaldevelopment process (subject of intervention) and carried out in real terms, exemplified by theIROP and ROP programs with respect to the full context of their implementation.

The investigation has led to following achievements:

• Systemization and synthetization of issues regarding methodology of evaluation researchconcerning specific interventions in regional development has enabled to place evaluation ofregional programs an applicable strand of research in socio-economic geography.

• Developed model for evaluation of regional programs is based on the concept of paradigmof regional development and allows to apply regional science and policymakers’ perspectivesIt serves a foundation to formulate requirements for more reliable evaluation studies.

• Paradigm of regional development is regarded overarching for scientific and practicalapproach. It allows to combine contemporary views on regional development process as aresult of intended changes evoked mainly (though not entirely) by public intevention. Thecomponents of paradigm evolve over time with changing values affecting perception of socialneeds and various theories on how to effectively mobilise developmental factors. The conceptallows to measure cumulative effects of similar interventions, even if they were implementedunder formally different programs.

.

Summary. Conclusions. Discussion.

• The model extends the scope of standard evaluation of regional programs, because allcomponents of paradigm are examined with respect to their validity.

• The model respects the information needs of policymakers that refer to impact analysis ofintervention and assessment of intrisinc characteristics of intervention. The model seeks fora balance between the strategic and operational requirements, when in real terms theexamination of intervention is given far more consideration.

• The model recognizes two basic needs i.e. deficits to be considered in the construct ofregional program: social and spatial needs of targeted groups and areas.

• The model considers that mechanisms of intervention are generally based on sound economictheories of regional development, but they are very ecclectic and often take form of individualconstructs designed by policymakers.

• Verification of the model carried out with three various evaluation designes showcases theirdifferent utility for purpose of evaluating regional programs. It singles out the theory-drivenevaluation characterized by the highest responsiveness in both cognitive and methodologicaldimensions of analysis. This design corresponds best with the requirements of the model,while the other give mixed results depending on dimension of analysis.

• It is recommended to re-orient the ToR’s evaluation questions towards identifying allcomponents of the model (especially the mechanism of exerting an impact by intervention),while putting less politcal pressure on the description of outcomes. Strategic ex-postevaluations should be carried out less frequently than they currently are, allowing to considerlonger time-series.

by Anna [email protected]

mailto:[email protected]