The Policy Process - Evaluation

What is evaluation?

• “a critical and detached look at objectives and how they are being met”

• “The systematic application of social research procedures for assessing the conceptualisation, design, implementation and utility of public programmes”

• “The process of forming a judgement on the value of a programme”

Policy evaluation

“The purpose of evaluation research is to measure the effects of a programme against the goals it set out to accomplish as a means of contributing to subsequent decision-making about the programme and improving future programming”

(Weiss, 1972, p.4)

Distinction between evaluation and

descriptive/analytical research

• Evaluation research is intended to be used by decision-makers

• The basic questions to be answered are set by policy-makers and administrators

• A statement of policy goals is a necessary element of the research

• The publication of evaluation research is not a foregone conclusion

How does evaluation research aid decision-making?

Helps them: • Reach conclusions about continuing a

policy; improving a policy; adding or withdrawing features of a policy

• Decide whether a similar policy should be introduced elsewhere

• Allocate resources between competing alternatives

• Accept or reject the theoretical assumption on which a policy is based

Formative v’s summative

evaluation• Formative evaluation produces information

that is fed back during the implementation of a programme

• Summative is done when a policy has been implemented – to assess overall effectiveness of a policy

• Both are to assess the impact of policy on real-world conditions….

Policy Impact

• Impact on the target situation or group

• Impact on others that those intended (spillover effects)

• Impact on future as well as immediate concerns

• Direct costs, in terms of resources devoted to programme

• Indirect costs, including loss of opportunity to do other things

Difficulties in making evaluative designs work

• Policy objectives are often amorphous• The gap between aims and

implementation• Many factors “extraneous variables”

affect particular outcomes• Policy makers often have no regard for

the evaluation – thus limiting the research design

Some policy related criticisms

• Even if strong effects are shown these don’t often impact on policy

• Evaluation research may validate a particular view of social problems by emphasizing certain outcomes as opposed to others

Limits of Public Policy that affect Evaluation

• If problems are defined in relative rather than absolute terms they they may never be resolved by pubic policy.

• Expectations can outrace the capabilities of government

• Policies that solve the problems of one group, can create problems for another group.

• Societal forces can not always be harnessed by government.

Labour’s Welfare to Work Programme

• New Deals Objectives

– To increase the long term employability of help young and long-term unemployed people, lone parents and disabled people into jobs; and

– Improve their prospects of staying and progressing in employment.

Labour’s Welfare to Work Programme

• New Deals Assumption

• …that welfare dependency and unemployment can be substantially reduced both by improving the employability of working age benefit recipients and by connecting them more proactively to the labour market

Labour’s Welfare to Work Programme

• New Deals Implementation

• Individuals who fail to take up a place or leave early without good cause will have sanctions levied against them via benefits

Labour’s Welfare to Work Programme

• New Deal Impact?• Overall statistics mask the uneven decline of

unemployment rate• The areas with the lowest unemployment rate

has the highest success at further reducing the figures

• The areas with the highest unemployment rate had the lowest success at further reducing the figures.

• No jobs to go to? Wasting resources?

Labour’s Welfare to Work Programme

• Initial Assessment

– Government clamed positive impact with unemployment figures at its lowest in 20 years by 2000.

– TUC said direct affects will not be known for longer time– Academics suggested that results were best with

programmes that were ‘selective, small scale and resource intensive’ – expansion could reduce success rate

– New Deals success is dependent upon continued economic growth and macroeconomic strategy

Mobile Phone Networks in Northern Ireland

• 1996 Interagency Working Group on Displaced Families established

• Establishment of Phone Network in Belfast

• Funding agreed by MBW/BRO (government funded)

• Hamilton 2001 evaluation seeks to examine the operation, impact and resourcing of MPN’s

Mobile Phone Networks in Northern Ireland

• Caution against a ‘performance target’ analysis of the networks but rather:

– See resourcing MPN like paying insurance – the benefits are only clear when an accident occurs.

– Traffic Calming Measures – tend not to be built until after an accident

Mobile Phone Networks in Northern Ireland

“The riot which didn’t occur, the houses that weren’t damaged or the life that wasn’t lost may reflect the true impact of the phone networks, but this, obviously, can never be proven.” (Hamilton, 2001:22)

Mobile Phone Networks in Northern Ireland

“Conversely, where there is tangible evidence in, for example, the number of displaced families, or the number of recorded riot situations, it is impossible to attribute this with any degree of certainty to the success of the MPN’s.” (Hamilton, 2001:22)

Mobile Phone Networks in Northern Ireland

• Although speculative – a cost benefit analysis of MPN proved favourable - £16,000pa …..

Mobile Phone Networks in Northern Ireland

• A senior officer in North Belfast told a phone holder that by defusing a potential riot through the use of the MPN his sub-division saved £18,000 as he didn’t have to call out the Mobile Support Unit that tackles street disturbances.

• Construction of 13 Peace walls was £1,927,000

• For Translink, a new bus costs between £120-150,000 and only the estimated value is recouped from the Compensation agency (between £7-£50,000)

Wilson’s Laws

• Harvard professor James Q. Wilson formulated 2 laws on social science research/evaluation on policy

• Wilson’s First Law – All policy interventions in social problems produce the intended effect – if the research is carried out by those implementing policies or their friends.

• Wilson’s Second Law – No policy interventions in social problems produces the intended effect – if the research is carried out by independent third parties, especially those sceptical of the policy