Monitoring and Evaluation Guidance Version 1.02 November 2008

Monitoring and Evaluation Guidance

This document contains guidance to assist you to develop your monitoring and evaluation plan.

Your organisation may already have detailed procedures in place for monitoring and evaluation. If so you should draw upon the strengths of your current systems and apply them to the management of your new project. However, you should be sure that your procedures are sufficient to meet the requirements of WEFO and the European Commission for the purposes of the delivery, monitoring and evaluation of your project. You should confirm this with your WEFO Project Development Officer.

Monitoring and Evaluation Guidance Version 1.02 November 2008 Contents 1. MONITORING..............................................................................................3

Introduction .........................................................................................3 Why Monitor? ......................................................................................3

2. INDICATORS AND FORECASTING ...........................................................4 Selecting Indicators ............................................................................4 Forecasting..........................................................................................5

3. MONITORING SYSTEMS ...........................................................................6 Databases ............................................................................................6 Recordkeeping ....................................................................................7

4. REPORTING................................................................................................7 Why Report?........................................................................................7 When and How Should a Project Report?.........................................7 Management Information....................................................................7 Verification...........................................................................................7

5. EVALUATION..............................................................................................8 Introduction .........................................................................................8 Why Evaluate?.....................................................................................8

6. EVALUATION PLAN ...................................................................................9 Setting Aims and Objectives..............................................................9 Who Should Evaluate and When? .....................................................9 Evaluation Methods ..........................................................................10

7. MANAGING AN EVALUATION.................................................................11 Terms of Reference for an Evaluation.............................................11 Involving Stakeholders .....................................................................13 Monitoring Progress .........................................................................13 Final Reports .....................................................................................14 Using Evaluation Results .................................................................14 Freedom of Information ....................................................................15

8. OBJECTIVES OF PROJECT EVALUATION ............................................15 Demonstrating the Need for your Project .......................................15 Evaluating Project Processes ..........................................................15 Evaluating Project Outcomes ..........................................................16 Reporting against Impact Indicators ...............................................16

Annex 1 – Evaluation Summary Sheet .......................................................17 Annex 2 – Reporting Jobs Created ............................................................19

2

Monitoring and Evaluation Guidance Version 1.02 November 2008 1. MONITORING Introduction Monitoring is the regular and systematic collection of data (information) which are then used to help manage a project. The data are usually numerical and relate to finances or to other indicators and are processed to measure progress, for example in relation to baselines and forecasts. This guidance covers the non-financial aspects of monitoring. Why Monitor? Monitoring is essential for your project to succeed. Projects which have good monitoring systems in place are:

• better managed. Monitoring increases the likelihood of a project delivering high quality outcomes on time;

• more likely to be able to explain the reasons behind project successes (and failures);

• in a position to share evidence of their good practice with others; • more likely to meet regulatory requirements; • in a better position to get continuation funding; and are • contributing to an assessment of how well the Structural Funds have

worked in Wales.

3

Monitoring and Evaluation Guidance Version 1.02 November 2008 2. INDICATORS AND FORECASTING Selecting Indicators You should select your indicators at an early stage of project planning by choosing from the relevant list of Priority-level indicators provided in the Operational Programme (OP). The Strategic Framework documents highlight the indicators which may be of particular relevance to your project. You should choose indicators which will help you to demonstrate progress in the implementation of your project’s objectives. Selection of indicators will need to be discussed and agreed with your WEFO Project Development Officer. There are two types of monitoring indicators: output and result ( further information on impact indicators can be found in Section 10). Outputs are the activities undertaken by the project and results are the direct consequence of the activity. Results follow from the activities and are the immediate effect of your project, and reflect the key aims of your project. Impacts are the consequences of the project beyond the immediate results and are recorded through evaluation.. Your WEFO Project Development Officer or a member of WEFO’s Research, Monitoring & Evaluation team (RME) will be able to advise on indicator selection. The indicators contained within the OP are not intended to cover the whole breadth of your project’s activity, but rather are a core set which relate to the Programme’s key aims. You may choose to develop additional indicators for your particular project; again, these should be chosen at the planning stage and further advice can be provided by WEFO if necessary. In addition, projects, particularly ESF projects, may wish to measure soft outcomes. These are changes such as improved self-esteem or better job search skills. Guidance on using soft outcomes can be found at: http://www.wefo.wales.gov.uk/resource/Soft_Outcomes_Leavers_Study_E7217.pdf

4

Monitoring and Evaluation Guidance Version 1.02 November 2008 Forecasting You will be required to provide WEFO with forecasts for the Priority-level indicators you have selected. Forecasts will be needed for set points throughout the lifetime of your project. Projects will be required, for certain indicators, to forecast progress against indicator subcategories, such as the gender or ethnicity of participants. These subcategories are called “category breakdowns”. The frequency of forecasts will be agreed with your WEFO Project Development Officer. For example, a project may need to provide quarterly forecasts. This forecast information, along with similar forecasts for project spend, forms your project’s “Delivery Profile”. You will be required to report against this Delivery Profile throughout project implementation. Forecasts should be set so that they are challenging but achievable. It helps if evidence is available on which to base forecasts: for example, data on the performance of other similar projects. Researching the context of your project can also provide useful evidence for forecasting. For example, if your project aims to provide business advice to enterprises within a certain sector, it may be useful to know the number of enterprises in the area covered by the project. The relationship between outputs and results should be considered when forecasting and reflected in the forecasts provided to WEFO. Forecasts must be agreed with your WEFO Project Development Officer during the development stage of your project. Data Protection Act 1998 (DPA) – Be aware that anyone processing personal information must comply with the DPA principles. Please see www.ico.gov.uk for further guidance

5

Monitoring and Evaluation Guidance Version 1.02 November 2008 3. MONITORING SYSTEMS

Developing a good monitoring system is an essential part of developing your project. The system you use should be suitable for your needs. It must be able to provide you with information to help you manage the project, as well as the information you need to report to WEFO. The first step should be to consider what information you will need to run the project and what information you are required to report to WEFO. You will then need to decide upon the type and scale of the monitoring system you require. The scale of the system should be in proportion to the size of your project. For example, a small project will not need a large and complicated database. In some cases a simple spreadsheet will suffice to keep track of progress. Databases Most projects will need some sort of database to hold the monitoring data. A very basic database could be set up in-house and would require little support but will have limited capabilities. Such a database does not provide administrative support and only very basic management information. You may need a bespoke database which can be used by many users and have support staff. An example Participant database has been provided by WEFO and includes an Excel spreadsheet containing examples of Excel formulas which you may find useful for calculating aggregate information for reporting to WEFO. An example Participant database including an Excel spreadsheet containing examples of Excel formulas which you may find useful for calculating aggregate information can be found at http://www.wefo.wales.gov.uk/default.asp?action=page&ID=2 Spreadsheets are straightforward to set up and are useful for recording the information from small projects or a small number of activities. They can also be used as an interim measure until a more permanent solution is found. They are not suitable when large amounts of information need to be stored.

6

Monitoring and Evaluation Guidance Version 1.02 November 2008 Recordkeeping It is important you ensure that sufficient records are kept to satisfy audit requirements. Further guidance on recordkeeping can be found at http://www.wefo.wales.gov.uk/default.asp?action=page&ID=2166 It is also important for projects to keep a record of Participant’s consent to pass detail to WEFO. Be aware that anyone processing personal information must comply with the Data Protection Act 1998 (DPA). Guidance on DPA can be found at www.ico.gov.uk 4. REPORTING Why Report? Reports are useful for you to inform stakeholders of the progress of your project. Reports should provide an early indication of what your project is delivering successfully or give an early warning of any difficulties. All Structural Fund supported projects are required to report progress to WEFO against a number of indicators. The data you provide to WEFO will be used by the Programme Monitoring Committee (PMC) and the European Commission to monitor Programme progress and will be used to inform Programme-level evaluations of progress against impact indicators. When and How Should a Project Report? At the project planning stage you will have developed a delivery profile and agreed a timetable for reporting progress against your indicators to WEFO. This will include the category breakdown information against which you will need to report. The frequency for reporting against your project's indicators will vary but will be agreed with WEFO when the project is approved. To report progress to WEFO against your project's indicators, you will need to log on to PPIMS and update your project's Delivery Profile. Management Information In addition to the monitoring data you are required to collect, you may consider collecting additional monitoring data to help with better project management. For example, you may wish to keep a database of lessons learned during the project that may be used to inform the planning or delivery of future projects. You may wish to keep a database of issues that occur which may affect the successful delivery of your project. Verification

7

Monitoring and Evaluation Guidance Version 1.02 November 2008 Examples of the types of evidence required by WEFO to verify your activities are outlined within the indicator definition guidance. Indicator definitions can be found at http://www.wefo.wales.gov.uk/default.asp?action=page&ID=2 Further guidance on project verification can be found at http://www.wefo.wales.gov.uk/default.asp?action=page&ID=2166 5. EVALUATION Introduction Evaluation is wider in scope than monitoring. It includes making an assessment of the overall achievements of a project. An evaluation addresses questions such as: • Have the objectives of the project been met? • How efficiently were the outputs and results achieved? and • What would have happened without the intervention? Why Evaluate? The main reason for evaluating your project is to assess whether it has achieved its objectives. Although monitoring information can help you make this assessment, it usually will not be enough to produce a rounded judgement of success. This is because the data usually do not address questions about quality of achievement or consider contextual factors which have helped or hindered achievement. For example, your monitoring data may tell you that your ERDF project has created 100 jobs but, some of these jobs may have been created even if your project had not gone ahead. Also, monitoring information will not tell much about the quality of these jobs. Evaluation permits an in-depth investigation of such issues. An evaluation will explain the reasons behind project successes and failures. This will enable you to share evidence of good practice with others. If some aspects of your project have not worked so well, evaluation can provide recommendations of what might work better next time. Your project-level evaluation will also contribute to an overall assessment of how well the Structural Funds have worked in Wales. For the 2007–2013 Structural Funds Programmes in Wales all project sponsors are required to undertake or commission evaluations of their projects (See the Monitoring and Evaluation Section of the Implementation chapter in the Operational Programmes). All projects awarded £2 million grant or more (ESF or ERDF) for a single project and all projects involved in

8

Monitoring and Evaluation Guidance Version 1.02 November 2008 implementing ERDF supported innovative or experimental actions and all projects identified as innovative under Article 7 of the ESF Regulation are required to have their project evaluated by an external independent contractor. If your project falls below the threshold for a mandatory external evaluation you may still wish to appoint an external contractor because of the independent perspective which can be provided. The question of who should evaluate is closely linked to the issue of size and scope of the evaluation. The size of your evaluation should be proportionate to the size of your project. Costs associated with undertaking an evaluation are eligible for Structural Fund assistance. 6. EVALUATION PLAN Setting Aims and Objectives When developing your project it is vital to put together an evaluation plan. The evaluation plan will outline what evaluation activity will be undertaken during the life of your project (including project closure). The plan will state when the evaluation will be undertaken, who will carry out the evaluation and what evaluation methods will be used. The evaluation plan will identify the evaluation questions to be addressed and will outline the scope and size of the proposed evaluation. The size of the evaluation should be proportionate to the size and risk of the project and will be agreed with the WEFO Project Development Officer at the development stage. The scope of evaluation will also vary, depending on the size of the project. For example, a smaller project might not undertake a complex impact evaluation, but a half day facilitated reflection. The cost associated with undertaking evaluation is eligible expenditure. Who Should Evaluate and When? All sponsors are required to undertake or commission evaluations of their projects. All projects awarded £2 million grant or more (ESF or ERDF) for a single project and all projects involved in implementing ERDF supported innovative or experimental actions and all projects identified as innovative under Article 7 of the ESF Regulation are required to have their project evaluated by an external independent contractor. If your project falls below the threshold for a mandatory external evaluation you may still wish to appoint external contractors because of the independent perspective they can provide. The question of who should evaluate is closely linked to the issue of size and scope of the evaluation. Do you have the resources and expertise internally to carry out the size and scope of evaluation you envisage? If not then you should consider contracting an external evaluator. The resource implications of deciding who should evaluate

9

Monitoring and Evaluation Guidance Version 1.02 November 2008 will consist mainly of staff’s skills and time for an internal evaluation and cost for an external evaluation, although contract management will require a time commitment too. Please do not underestimate the resource intensity for contract management. Another factor to consider is risk. Projects regarded as high-risk and/or high-profile will require a particularly thorough evaluation which may be resource intensive. Projects which are innovative or pilot in nature or where an evaluation has a learning or participatory element will also require more resource intensive evaluation. Evaluation Methods Evaluation methods can be broken down into two groups: methods for collecting data and methods for analysing data. The selection of evaluation methods has resource implications. If a project requires a particular evaluation method that is expensive or time-consuming to implement (e.g. face-to-face interviews) then the resource implications of this method may be balanced against the limitations of using a less resource-intensive method (such as telephone interviews or a postal survey). The amount of data to be analysed also has resource implications. Methods for collecting data This covers the collection of primary and secondary data. The methods you select should be appropriate for addressing the evaluation questions you have identified in your plan. The methods should take into consideration the size and scope of the evaluation and if undertaken internally, the experience and expertise of whoever will be undertaking the evaluation. The main methods of collecting data are: • Observation – a qualitative method, whereby the activities of project staff and/or participants are observed; • Focus groups – a qualitative method, whereby group discussions are co-ordinated by facilitator; and • Interviewing –un-structured or semi-structured questions provide

qualitative data, closed questions provide quantitative data. Interviews can be undertaken in groups; face to face, by telephone, through the post, or online; and

• Literature reviews – data are gathered from already existing sources (secondary data).

When collecting data it is important to consider the appropriateness of your sample in terms of size and make up. For example, if your evaluation aims to produce findings which are to be generalized, the sample will need to be random, but representative of the general population, and of sufficient size that findings can be considered reliable. However, if the evaluation aims to provide illustrative case studies of certain practices or contexts, then the your

10

Monitoring and Evaluation Guidance Version 1.02 November 2008 sample will be specifically selected for this purpose and you will not need a large sample. Analysing Data How you analyse your data should be considered when deciding the methods, size and scope of your evaluation. The main methods for analysing data are: • Quantitative methods –statistics are used to provide the characteristics

of a sample and, if there are sufficient data, to derive conclusions about the performance of your whole project..

• Qualitative methods – patterns and themes emerging from interview transcripts, recordings or photographs can be categorized (or coded) to provide an analytical description of a sample and the key concepts emerging for the whole project.

7. MANAGING AN EVALUATION The main principles of managing an evaluation are essentially the good project management principles that you will already be using in the management of your Structural Funds project. These include:

• Developing clear objectives for the evaluation; • Deciding on the resources needed to do the work; • Planning the various stages of the evaluation; • Involving all relevant stakeholders; and • Monitoring progress against your plan.

Guidance on good practice in Structural Funds project management can be found at: http://www.wefo.wales.gov.uk/default.asp?action=page&ID=1520 Terms of Reference for an Evaluation You will need to write a Terms of Reference (or specification) for your evaluation, whether undertaken in-house or externally contracted.

A sample Terms of Reference can be found at: http://www.wefo.wales.gov.uk/default.asp?action=page&ID=2 A good Terms of Reference for an evaluation will include the following information: 1. Introduction - an introduction to the organisation, to the project, and to

the proposed evaluation, including details of how the evaluation is important to the project sponsor.

2. Background / requirement for evaluation - clear aims and objectives of the evaluation.

11

Monitoring and Evaluation Guidance Version 1.02 November 2008 3. Method - two different approaches could be employed: i) propose a

detailed methodology and allow the contractor to cost the work; or ii) provide an outline of your thinking and a cost limit to allow contractors to propose a methodology. There are risks and benefits of both approaches. WEFO will provide assistance with choosing the most appropriate approach for your project.

4. Reports – provide details of the reports you require. This may include inception (a detailed work plan), interim, draft, and final reports and for an externally contracted evaluation and a possible requirement that the contractor produce a stand-alone executive summary of the report (no more than approximately five sides long).

5. Dissemination guidance. see “Using Evaluation Results” below. 6. Timetable / milestones – a realistic timescale, including important

milestones, for the evaluation should be set. 7. Reference to the Data Protection Act 1998 (DPA) – Be aware anyone

processing personal information must comply with the DPA principles.

See www.ico.gov.uk for further guidance.

The Terms of Reference for an externally contracted evaluation will also include: 8. Contract award criteria - explicit criteria for awarding the contract (for

example on the contractor’s understanding of issues, appropriateness of proposed methods, value for money, etc.) and a weighting for these criteria. See terms of reference template and sample scoring sheet for further guidance.

9. Contract management - identify a project contact with responsibility for liaising with the suppliers who tender for the evaluation and with the contractor who is appointed to carry out the evaluation. You will need to consider the frequency and means of communication between contractor and project contact, and those who will oversee the evaluation (e.g. an evaluation steering group).

10. Cost of evaluation - you can provide contractors with a Terms of Reference that details the method you wish them to follow, and let them provide you with quotes for the cost of following the methodology; or, you can give an indicative budget and allow suppliers to propose their own methodology that meets the evaluation’s objectives within this budget.

The chosen contractors should be independent of the project and should not include any of the project stakeholders. They should be suitably qualified and experienced. The tender they submit should demonstrate an adequate understanding of the topic, the context and the research issues. The proposed methodology should be workable and cover all key research aims and objectives. It may be useful to supplement the Terms of Reference with a more detailed timetable which lists all the tasks to be completed and assigns responsibility

12

Monitoring and Evaluation Guidance Version 1.02 November 2008 for each of these. If you are managing an external evaluation, the tasks will relate to the contract management aspects. The contractors should construct a timetable based around the work they have been contracted to undertake. You should check that this meets with your expectations. If you are commissioning a large evaluation you will need a detailed timetable as part of an inception report. An inception report is a detailed work plan for the evaluation and this can be used as a check for you to ensure you are content with the evaluator’s approach to the work. Procurement Process If you are commissioning an external evaluation you will also need to plan the procurement process. When planning this stage you will need to allow sufficient time and resources for the following tasks:

• Advertising the contract and allowing suppliers a reasonable amount of time to submit a tender; and

• Assessing tenders, which may include: o Notifying all suppliers of final decision; and o If requested, providing feedback to unsuccessful suppliers.

Involving Stakeholders An evaluation steering group can be an effective way of involving stakeholders early in the evaluation process. A steering group can also help you make key decisions on different aspects of the evaluation e.g. what the evaluation should cover. If your project already has a project board, your evaluation steering group can be a sub-group of the main project board. Your steering group should include those who will have an interest in the final results. Individuals within your organisation who have a good understanding of different aspects of the project (including the monitoring data) should also be included as they can provide the evaluators with useful contextual information. The frequency of steering group meetings will depend on the size and complexity of the evaluation. For small evaluations it may not be necessary to set up a formal steering group but you should still take care to involve stakeholders e.g. through email updates or informal meetings. Monitoring Progress To ensure that your evaluation is on track to achieve its objectives on time you should have regular meetings with the evaluation team whether these are external contractors or colleagues within your organisation. At these meetings you should discuss progress against the timetable and whether issues have arisen which may affect the quality of the evaluation or may hinder progress.

13

Monitoring and Evaluation Guidance Version 1.02 November 2008 For larger evaluations it is sometimes necessary to have regular progress reports. An interim report can be used to present initial findings to the steering group and is a good way of communicating progress to stakeholders. Less formal progress reports can take the form of brief email updates. Draft Final Reports are also a useful tool for ensuring that the requirements of the terms of reference or specification have been met. Final Reports All projects will be required to submit a copy of their final evaluation report to WEFO. A sample of these evaluations will be summarised by WEFO to provide a useful reference of project evaluation practice. From time to time WEFO will wish to share good practice identified with other projects, but will always seek permission first from the project managers. WEFO’s project evaluation summary sheet can be found at Annex 1. One possible structure for an evaluation report is:

• Stand alone executive summary of findings, conclusions and recommendations;

• Background to the project and evaluation, including a review of the broader economic or social context relevant to project activity;

• Methodology – to cover sampling, data collection methods and analysis methods. These should be in sufficient detail so that the reader is able to understand the evidence base of the research. It should detail potential biases and include a discussion of the strengths and weaknesses of the evidence base;

• Assessment of achievement of project objectives, including the evidence;

• Conclusions and recommendations; and • Technical Annex – include technical details and research tools used in

the evaluation. It is important that the final report is written in language accessible to non-specialists and is self-explanatory to those who are unfamiliar with your project. Using Evaluation Results The main audience for your evaluations are yourselves, as project sponsor, the project board, people within your organization identified stakeholders (particularly those involved at an earlier stage of the evaluation), evaluation participants, similar projects and WEFO. You will need to ensure that your key stakeholders are aware of your evaluation results. You may need to use a variety of dissemination methods, including, but not limited to: bulletins; published papers; meetings; seminars; conferences; workshops and online. To ensure that learning opportunities presented by evaluations are taken up it may be appropriate to conduct a review some time following the completion of the evaluation to consider progress against any recommendations made in an

14

Monitoring and Evaluation Guidance Version 1.02 November 2008 evaluation. You may wish also to share this learning with other projects to ensure that good practice is shared. Freedom of Information Completed reports, by public authorities, at any stage of an evaluation, are subject to the Freedom of Information (FOI) regulations. Further guidance on the F0I regulations is at: http://www.justice.gov.uk/whatwedo/freedomofinformation.htm 8. OBJECTIVES OF PROJECT EVALUATION The following outlines the key objectives that can be covered in your project’s evaluation. Demonstrating the Need for your Project For your project to be eligible for Structural Fund assistance its aims and objectives must fit in with the Operational Programme and Strategic Framework documents. As part of this you will need to demonstrate the evidence for the need for your project. You should use as much existing background data and information as you can gather when deciding your aims and objectives. It is possible that there will be a lack of information on which to base your aims and objectives. In such instances you may decide to collect, or commission the collection of, data to fill the gap. Large projects can go a step further and carry out an evaluation to identify the need for the project and help set its aims and objectives. A project may also carry out research to see if the projects aims, objectives and plans are realistic and deliverable (this is called a feasibility study). At this stage of the project it is also important that baseline data are recorded. Progress cannot be assessed meaningfully without baseline information in place. Evaluating Project Processes This covers issues such as: • Whether and to what extent the project is still relevant to its external

context, for example, its socio-economic environment; • Progress to date against key output, result and financial indicators;

15

Monitoring and Evaluation Guidance Version 1.02 November 2008 • Whether the project is working well as a mechanism for delivering its

aims; • How efficient are the project processes; and • Recommendations to address any issues found. Process evaluation questions and suggested methodologies can be found at: http://www.wefo.wales.gov.uk/default.asp?action=page&ID=2 Evaluating Project Outcomes This covers issues such as: • Did the project achieve, or is on course to achieve, its objectives? • If not, why not? • What has worked well/not so well? • What is the impact of the project likely to be? • What is the external context? • How has this changed and how has it affected the project’s outcomes? • How efficient were the project processes? • Could the objectives have been achieved in another way? • Have recommendations from any previous evaluations been

implemented? If so, what effect have they had? ERDF and ESF impact evaluation questions and suggested methodologies can be found at: http://www.wefo.wales.gov.uk/default.asp?action=page&ID=2 Reporting against Impact Indicators If your project is aligned with a particular Strategic Framework you should use the evaluation questions contained within that Strategic Framework document. These questions are based on the Priority-level impact indicators and should form the basis for the evaluation of your project. The Operational Programme shows the logical flow from output to result through to impact. You should follow this logical flow with respect to your selection of impact indicators.

16

Monitoring and Evaluation Guidance Version 1.02 November 2008 Annex 1 – Evaluation Summary Sheet

Project name: Project sponsor: Project reference: Programme: Priority (Theme): Strategic Framework: 1. Scope 1.1 What are the objectives of the project? 1.2 What are the objectives of the evaluation? 1.3 What are the limitations of the evaluation? E.g. cost, limited response, timescale. 1.4 Does the evaluation cover both successful and unsuccessful aspects of the project? 1.5 Was the evaluation: undertaken internally; externally contracted; or a mixture of both? 1.6 When, in relation to the project like cycle, was the evaluation undertaken? 2. Methodology 2.1 What quantitative methods were used to address the evaluation questions? 2.1a What were the sources of data for these? 2.1b Are the research tools included in the report? If so, provide a brief description. 2.2 What qualitative methods were used to address the evaluation questions? 2.2a What were the sources of data for these? 2.2b Are the research tools included in the report? If so, provide a brief description. 2.3 Detail any ‘mixed methods’ approaches used. 2.3a What were the sources of data for these? 2.3b Are the research tools included in the report? If so, provide a brief description.

17

Monitoring and Evaluation Guidance Version 1.02 November 2008

2.4 What are the limitations of the data (e.g. non-response)? How has the potential bias of these been accounted for? 3. Analysis 3.1 What analysis has been undertaken? E.g. inferential statistics, qualitative analysis. Provide details against each evaluation objective. 3.2 Provide descriptions of where findings have been triangulated (i.e. multiple sources of evidence obtained to support conclusions) 3.3 What were the findings of the evaluation? 3.4 What were the recommendations of the evaluation? 3.5 To what extent has each evaluation objective been met? 4. Report 4.1 Is the report written in accessible language (i.e. could be understood by a non-specialist and self-explanatory)? 4.2 Provide brief details of the reports structure (for example, a list of section headings).

18

Monitoring and Evaluation Guidance Version 1.02 November 2008

Annex 2 – Reporting Jobs Created This note provides guidance for project sponsors on how to classify jobs as project outputs and how these outputs should be reported to WEFO as monitoring and evaluation data. The guidance gives definitions and outlines details of WEFO’s evidence requirements and explains how the monitoring and evaluation indicator data should be reported to WEFO. The form and frequency of reporting project monitoring data will be agreed with the project’s WEFO Project Development Officer (PDO) during project development. The guide does not replace the guidance and agreement given and made by the PDO. Where it is the intention of a project to create new jobs only new permanent posts should be counted as being created by the Structural Fund intervention. Definitions of the job creation indicators and the data required to evidence them are at Annex A. Permanent jobs are posts that will be created on an enduring basis; they can be expected to have no finite life-time. Only the actual posts should be counted, there should be no estimate for the number of people who may occupy the posts over time. Seasonal jobs may be counted if the job is expected to recur indefinitely. Here the project should report the proportion of the year worked, in addition to number of hours per week. Project management/administration jobs within the Structural Fund project cannot be included. The data collected should include detail of the post code and Unitary Authority in Wales (including “Outside of Wales”) in which the majority of the jobs are located. To help identify the quality of the jobs all jobs should be categorised into five salary bands (£14,999 and below; £15,000 to £19,999; £20,000 to £24,999, £25,000 to £34,999 and £35,000 and above). Jobs should be reported as Full Time Equivalents (FTEs), based on a 30 hour week. A post which involves working 30 hours or more per week is 1 FTE. If a job is not full-time, then the hours worked each week are divided by 30 to give the proportion of FTE represented. For example, 18 hours per week would be (18/30 =) 0.6 FTE; 15 hours per week would be (15/30 =) 0.5 FTE.

19

Monitoring and Evaluation Guidance Version 1.02 November 2008 Projects are required to report the number of FTEs against the salary band (see above) that reflects the FTE salary paid for that post. The following examples illustrate how this should be done:

• a project creates two part time jobs of 15hrs, each paying £10k. The first of 15hrs at £10k is 0.5 FTE within the band £20,000 to £24,999 (i.e. £10k/0.5FTE = £20k); the second of 15 hrs at £10k is also 0.5 FTE within the band £20,000 to £24,999 (i.e. £10k/0.5FTE = £20k). 1 FTE within the band £20,000 to £24,999 should be reported. • a project creates two part time jobs each of 15hrs one paying £10k and

one paying £15k: The first of 15hrs at £10k is 0.5 FTE within the band “£20,000 to £24,999” (i.e. £10k/0.5 FTE = £20,000), and the second of 15 hrs at £15k is 0.5 FTE within the band “£25,000 to £34,999” (i.e. £15k/0.5 FTE = £30,000). • a project creates two part time jobs, one of 10hrs at £10k and one of

13hrs at £16k: The first of10 hrs at £10k is 0.33 FTE within the band “£25,000 to £34,999” (£10,000/0.33 = £30.3k) and The second of13 hrs at £16k is 0.43 FTE within the band “above £35,000” (£16, 000/0.43 = £37.2k).

20

Monitoring and Evaluation Guidance Version 1.02 November 2008

ANNEX A Definitions: Gross and Net Jobs Created Gross jobs created are the total number of permanent jobs achieved through Structural Fund activity (irrespective of the proportion of the project funded by Structural Funds) and has not been adjusted for deadweight, displacement, leakage and multiplier effects. All the jobs are achieved through Structural Fund assistance or financial support and did not exist prior to the Structural Fund activity. They do not include jobs which have been relocated. Evidence for gross jobs created will include personnel records showing the increase in headcount and any other documents showing that the jobs relate directly to the project Net jobs created are permanent posts achieved through Structural Fund activity and adjusted for deadweight, displacement, leakage and multiplier effects. They are achieved through Structural Fund assistance or financial support and did not exist prior to the Structural Fund activity. Net jobs are evidenced at the evaluation stage of a project. Guidance on impact assessment and an explanation of these terms can be found in: English Partnerships (2004) Additionality Guide: A Standard Approach to Assessing the Additional Impact of Projects, http://www.englishpartnerships.co.uk/communitiespublications.htm Further guidance can be found in: European Commission (2007) Working Document No 6: Measuring Structural Funds Employment Effects.

21