Creating Robust Monitoring Systems

Dr Susan DenmanHead of Research, Monitoring & Evaluation

Open Days Workshops “Managing Regional Development”

28-29 September 2004Brussels

Creating Robust Monitoring Systems

How can we ensure that we get high quality data which are useful for programme management and evaluation purposes?

www.wefo.wales.gov.uk

Structure of Presentation

• Welsh context

• Definitions

• Issues at design and planning stages

• Targeting projects for quality

• Issues at data collection and entry stages

• Reporting

www.wefo.wales.gov.uk

www.wefo.wales.gov.uk

The Welsh Context (1)

• Three main programmes:Objective 1

Objective 2 Objective 3

• Four Community Initiatives: INTERREG IIIA, URBAN II, LEADER+ and

EQUAL

www.wefo.wales.gov.uk

The Welsh Context (2)

• Total European Funding of €2,180 million

• Total value of €4,675 million (including private sector match funding)

Largest programme is Objective 1

Main aims:

• to create 43,000 net additional jobs

• to contribute to raising the Gross Domestic Product (GDP)

• reduce economic inactivity

www.wefo.wales.gov.uk

ED&T Group of Welsh Assembly Government

WEFO

ERDFINTERREG

Project Management Policy and Strategy

ESFEAGGF

FIFGLEADER+

EQUAL

LCDURBAN

Finance and Corporate Services

Paying Authority

Finance Payments Article 4 Article 10 IT

RM&EPolicy

BranchCommunic

ationsAssemblyBusiness

RM&E Research, Monitoring & Evaluation Branch

Monitoring and Reporting

4 staff: 1 researcher,3 administrators

Responsibilities:PMC Papers

Annual Monitoring Business PlansAnnual Implementation Reports

Local Partnership Reports

Evaluation

1 researcher,2 research and

evaluation specialists

Responsibilities:Data Quality

Programme Level EvaluationM&E advice and guidance

Head of Branch

Corporate

1 administrator, 1 researcher

Responsibilities:Training

DisseminationSynthesis Papers

Ex-Post EvaluationClosure Reports

Definitions

Monitoring is the regular and systematic examination of the resources, activities and results to inform management decisions

Audit is the verification of the legality and regularity of the implementation of resources.

Evaluation is a broad concept concerning the judgement made on the value of the initiative, which can encompass the needs which have to be met and the effects produced by it. Evaluation concerns not only outcomes but the processes by which they have been produced.

www.wefo.wales.gov.uk

Design and Planning

• Consider monitoring and evaluation at outset• start with ‘what’ not ‘how’• involve users from start• choose indicators with care• ensure hierarchy of indicators is coherent• build capacity

www.wefo.wales.gov.uk

Indicators

Helps if they are:• clearly linked with goal, objectives or targets• cover monitoring and evaluation needs • measurable on a regular basis using reliable

methods• provide clear and simple information• measurable against baselines• congruent with other monitoring indicator

systems

www.wefo.wales.gov.uk

Data Provided by Projects

Projects should:• understand purpose of centrally collected

data• collect other useful data• have access to training, support and

guidance• be clear on definitions• be checked for quality of their plans

www.wefo.wales.gov.uk

Project Sponsor

Gateway

Project development

Submission of application

Project appraisal

projectapproval

Delivery and review

Payments Audit

Closure

WEFO guidance

Advice on prioritisation from Thematic Advisory

Groups

WEFO advise

WEFOappraise

Advice from thematic groups, cross-cutting theme advisors and local partnerships

WEFO monitor

Advice from local partnerships and project officers

Assistance from local partnerships and project officers

Support and aftercare from local

partnerships and project officers

The Project Lifecycle and Monitoringwww.wefo.wales.gov.uk

Data Collection and Storage

• timing and methods• dealing with inaccuracies, gaps, double

counting and under-reporting • designing a good IT system • testing and evaluating the whole monitoring

system• management checks of data entry• liaison with global grants/grant schemes

www.wefo.wales.gov.uk

Use of data

• involve users in report design• co-ordinate release of data• ensure users understand data limitations • ensure that definitions are known and

understood by all• analyse the data collected• help with data interpretation

www.wefo.wales.gov.uk