tm evaluate effectiveness of assessment system refined€¦ · the association of southeast asian...

Evaluate the effectiveness of an assessment system

D1.HRD.CL9.07 D1.HHR.CL8.08

Trainee Manual

Project Base William Angliss Institute of TAFE 555 La Trobe Street Melbourne 3000 Victoria Telephone: (03) 9606 2111 Facsimile: (03) 9670 1330

Acknowledgements Project Director: Wayne Crosbie Chief Writer: Alan Hickman Subject Writer: Quentin Derrick Project Manager/Editor: Alan Maguire DTP/Production: Daniel Chee, Mai Vu, Kaly Quach

The Association of Southeast Asian Nations (ASEAN) was established on 8 August 1967. The Member States of the Association are Brunei Darussalam, Cambodia, Indonesia, Lao PDR, Malaysia, Myanmar, Philippines, Singapore, Thailand and Viet Nam.

The ASEAN Secretariat is based in Jakarta, Indonesia.

General Information on ASEAN appears online at the ASEAN Website: www.asean.org.

All text is produced by William Angliss Institute of TAFE for the ASEAN Project on “Toolbox Development for Front Office, Food and Beverage Services and Food Production Divisions”.

This publication is supported by the Australian Government’s aid program through the ASEAN-Australia Development Cooperation Program Phase II (AADCP II).

Copyright: Association of Southeast Asian Nations (ASEAN) 2013.

All rights reserved.

Disclaimer

Every effort has been made to ensure that this publication is free from errors or omissions. However, you should conduct your own enquiries and seek professional advice before relying on any fact, statement or matter contained in this book. The ASEAN Secretariat and William Angliss Institute of TAFE are not responsible for any injury, loss or damage as a result of material included or omitted from this course. Information in this module is current at the time of publication. Time of publication is indicated in the date stamp at the bottom of each page.

Some images appearing in this resource have been purchased from stock photography suppliers Shutterstock and iStockphoto and other third party copyright owners and as such are non-transferable and non-exclusive. Clip arts, font images and illustrations used are from the Microsoft Office Clip Art and Media Library. Some images have been provided by and are the property of William Angliss Institute.

Additional images have been sourced from Flickr and SXC and are used under Creative Commons licence: http://creativecommons.org/licenses/by/2.0/deed.en

File name: TM_Evaluate_effectiveness_of_assessment_system_refined.docx

© ASEAN 2013 Trainee Manual


Table of contents

Introduction to trainee manual ........................................................................................... 1

Unit descriptor................................................................................................................... 3

Assessment matrix ........................................................................................................... 5

Glossary ........................................................................................................................... 7

Element 1: Plan the evaluation of the assessment system ................................................ 9

Element 2: Undertake the evaluation of the assessment system .................................... 31

Element 3: Prepare an evaluation report ......................................................................... 45

Presentation of written work ............................................................................................ 63

Recommended reading ................................................................................................... 65

Trainee evaluation sheet ................................................................................................. 67

Trainee self-assessment checklist .................................................................................. 69

Introduction to trainee manual


Evaluate the effectiveness of an assessment system 1


To the Trainee

Congratulations on joining this course. This Trainee Manual is one part of a ‘toolbox’ which is a resource provided to trainees, trainers and assessors to help you become competent in various areas of your work.

The ‘toolbox’ consists of three elements:

• A Trainee Manual for you to read and study at home or in class

• A Trainer Guide with Power Point slides to help your Trainer explain the content of the training material and provide class activities to help with practice

• An Assessment Manual which provides your Assessor with oral and written questions and other assessment tasks to establish whether or not you have achieved competency.

The first thing you may notice is that this training program and the information you find in the Trainee Manual seems different to the textbooks you have used previously. This is because the method of instruction and examination is different. The method used is called Competency based training (CBT) and Competency based assessment (CBA). CBT and CBA is the training and assessment system chosen by ASEAN (Association of South-East Asian Nations) to train people to work in the tourism and hospitality industry throughout all the ASEAN member states.

What is the CBT and CBA system and why has it been adopted by ASEAN?

CBT is a way of training that concentrates on what a worker can do or is required to do at work. The aim is of the training is to enable trainees to perform tasks and duties at a standard expected by employers. CBT seeks to develop the skills, knowledge and attitudes (or recognise the ones the trainee already possesses) to achieve the required competency standard. ASEAN has adopted the CBT/CBA training system as it is able to produce the type of worker that industry is looking for and this therefore increases trainees chances of obtaining employment.

CBA involves collecting evidence and making a judgement of the extent to which a worker can perform his/her duties at the required competency standard. Where a trainee can already demonstrate a degree of competency, either due to prior training or work experience, a process of ‘Recognition of Prior Learning’ (RPL) is available to trainees to recognise this. Please speak to your trainer about RPL if you think this applies to you.

What is a competency standard?

Competency standards are descriptions of the skills and knowledge required to perform a task or activity at the level of a required standard.

242 competency standards for the tourism and hospitality industries throughout the ASEAN region have been developed to cover all the knowledge, skills and attitudes required to work in the following occupational areas:

• Housekeeping

• Food Production

• Food and Beverage Service


2


• Front Office

• Travel Agencies

• Tour Operations.

All of these competency standards are available for you to look at. In fact you will find a summary of each one at the beginning of each Trainee Manual under the heading ‘Unit Descriptor’. The unit descriptor describes the content of the unit you will be studying in the Trainee Manual and provides a table of contents which are divided up into ‘Elements’ and ‘Performance Criteria”. An element is a description of one aspect of what has to be achieved in the workplace. The ‘Performance Criteria’ below each element details the level of performance that needs to be demonstrated to be declared competent.

There are other components of the competency standard:

• Unit Title: statement about what is to be done in the workplace

• Unit Number: unique number identifying the particular competency

• Nominal hours: number of classroom or practical hours usually needed to complete the competency. We call them ‘nominal’ hours because they can vary e.g. sometimes it will take an individual less time to complete a unit of competency because he/she has prior knowledge or work experience in that area.

The final heading you will see before you start reading the Trainee Manual is the ‘Assessment Matrix’. Competency based assessment requires trainees to be assessed in at least 2 – 3 different ways, one of which must be practical. This section outlines three ways assessment can be carried out and includes work projects, written questions and oral questions. The matrix is designed to show you which performance criteria will be assessed and how they will be assessed. Your trainer and/or assessor may also use other assessment methods including ‘Observation Checklist’ and ‘Third Party Statement’. An observation checklist is a way of recording how you perform at work and a third party statement is a statement by a supervisor or employer about the degree of competence they believe you have achieved. This can be based on observing your workplace performance, inspecting your work or gaining feedback from fellow workers.

Your trainer and/or assessor may use other methods to assess you such as:

• Journals

• Oral presentations

• Role plays

• Log books

• Group projects

• Practical demonstrations.

Remember your trainer is there to help you succeed and become competent. Please feel free to ask him or her for more explanation of what you have just read and of what is expected from you and best wishes for your future studies and future career in tourism and hospitality.

Unit descriptor



Unit descriptor


This unit deals with the skills and knowledge required to Evaluate the effectiveness of an assessment system in a range of settings within the hotel and travel industries workplace context.

Unit Code:

D1.HRD.CL9.07 D1.HHR.CL8.08

Nominal Hours:

40

Element 1: Plan the evaluation of the assessment system

Performance Criteria

1.1 Identify the purpose and role of the evaluation

1.2 Define the assessment system

1.3 Identify the needs of the stakeholders

1.4 Identify and obtain resources to enable the evaluation

1.5 Develop an evaluation plan

1.6 Determine the evidence that needs to be captured during the evaluation

1.7 Define the criteria for determining the effectiveness of the assessment system

1.8 Identify cost-effective methods for capturing and analysing evaluation data

1.9 Develop tools to capture identified evidence regarding the effectiveness of the assessment system

Element 2: Undertake the evaluation of the assessment system


2.1 Trial the data gathering tools and techniques

2.2 Revise the data gathering tools and techniques on the basis of trials conducted

2.3 Collect the identified evidence in accordance with the approved evaluation plan

2.4 Store data

2.5 Analyse the data

Unit descriptor

4


Element 3: Prepare an evaluation report


3.1 Produce a written evaluation report

3.2 Distribute report to stakeholders for comment

3.3 Make a verbal presentation to support the report

3.4 Gather feedback on the report

3.5 Revising draft recommendations on the basis of feedback received

3.6 Determine action to be taken as a result of the evaluation

Assessment matrix



Assessment matrix

Showing mapping of Performance Criteria against Work Projects, Written Questions and Oral Questions

The Assessment Matrix indicates three of the most common assessment activities your Assessor may use to assess your understanding of the content of this manual and your performance – Work Projects, Written Questions and Oral Questions. It also indicates where you can find the subject content related to these assessment activities in the Trainee Manual (i.e. under which element or performance criteria). As explained in the Introduction, however, the assessors are free to choose which assessment activities are most suitable to best capture evidence of competency as they deem appropriate for individual students.

Work Projects

Written Questions

Oral Questions


1.1 Identify the purpose and role of the evaluation 1.1 1 1

1.2 Define the assessment system 1.2 2 2

1.3 Identify the needs of the stakeholders 1.3 3 3


1.4 4 4

1.5 Develop an evaluation plan 1.5 4 5


1.5 5 6

1.7 Define the criteria for determining the effectiveness of the assessment systemr

1.5 6 7


1.6 7 8


1.7 8 9

Assessment matrix

6


Work Projects

Written Questions

Oral Questions


2.1 Trial the data gathering tools and techniques 2.1 9 10


2.1 10 11


2.2 11 12

2.4 Store data 2.2 12 13

2.5 Analyse the data 2.2 13 14


3.1 Produce a written evaluation report 3.1 14 15

3.2 Distribute report to stakeholders for comment 3.2 15 16


3.3 16 17

3.4 Gather feedback on the report 3.4 17 18


3.5 18 19


3.6 19 20

Glossary



Glossary

Term Explanation

Allocate To assign or allot

Analyse To examine in detail

Assessor Someone who assesses

Criteria A standard by which something can be judged

Confirm To establish the ruth, validity or accuracy

Data Items of information

Determine To decide or conclude

Discontinue To cease or stop

Evaluation plan A plan that checks or judges the effectiveness of something

Evidence Facts

Executive summary A short abstract or compilation of the main parts of a report

Feedback Reaction or response to something

Identify To recognise or establish

Integrity The state of being whole and entire

Liaise Work with

Method Way of doing or completing something

Modify Alter or change

Outcome The result

Purpose The reason

Quantify To express the quantity of

Glossary

8


Term Explanation

Ratio The relation between two proportions

Record To set down or register

Reliability Able to be depended on

Revise To amend or alter

Stakeholder An individual or group who has an interest in something






Introduction

Determining why an evaluation is being conducted is the most important first step in beginning an evaluation project. The purpose and role of the evaluation need to be clear, specific and communicated effectively to all stakeholders. There are many different reasons why an evaluation of an assessment system may be conducted and these can range from an investigation of one part of the system to an evaluation of the assessment outcomes as a whole. The purpose of the evaluation should also reflect the organisation’s needs to make sure that the evaluation is useful and effective.

Define the purpose of the evaluation

Identifying the reason for conducting an evaluation is as important as identifying the people who will use the evaluation or stakeholders who may be a part of the consultation process. Both aspects of the evaluation act as a foundation for evaluation planning, focus, design, and interpretation and use of results. The purpose of an evaluation influences the identification of stakeholders for the evaluation, selection of specific evaluation questions, and the timing of evaluation activities. It is important that the assessment system evaluators are transparent about the intended purposes of the evaluation. If the evaluation results are to be used to determine if the assessment system will be continued or not, all stakeholders should be informed about this from the outset.

To develop a clear evaluation purpose, the evaluation team should gather feedback from key stakeholders who are involved in the process or are the sponsors of the evaluation, to identify the possible multiple purposes for the evaluation. The first step is to consider which stakeholders are interested in an evaluation of the assessment system. This could involve many different people but their input and needs are very important. The next step is to align the stakeholders’ interests with what they are requesting to be evaluated. After this stage, it is necessary to determine what the evaluation results will be used for and, finally, the evaluation team should create a purpose statement that is relevant to all stakeholders and what they want from the evaluation.


10


Below is a table that could be used to develop purpose statements for the evaluation

Stakeholders interested in the evaluation

What is to be evaluated?

What will the results be used for?

Evaluation purpose statement

Once the chart has been completed, any repeated or related purpose statements should be combined and finally an overall single purpose statement should be developed.

Evaluation Purpose Statement:

Evaluation Purpose Statement Examples

There are many different reasons why an evaluation of an assessment system may be conducted but they should be aligned to specific organisational needs. Some examples of evaluation purposes may be related to:

• Quantifying the number of staff who have participated in assessment, including separation of candidates into ‘Pass Competent’ and ‘Not Yet Competent’ categories

• Identifying the training topics, courses, programs and areas in which learners were subject to assessment for the period in question

• Determining what use will be made of the final evaluation

• Identifying the qualifications and/or experience and suitability of assessors who were engaged as part of the assessment process.

Activity: In pairs write down as many other purposes you can think of for conducting an evaluation of the assessment system. Use the chart above to help you.





Introduction

In order to determine what will be evaluated in an assessment system, it is first necessary to define the assessment system. There are many different aspects to the assessment system that will need to be outlined in order for the purpose of the evaluation to be aligned with the different parts of the assessment system. The definition of the system should be clear, concise, complete and transparent.

Defining the assessment system

The definition of the assessment system may include:

• An ability to determine the required competency levels and qualifications of assessors

• Policies and procedures for obtaining, updating and storing records and information of candidates and assessors. This may include version control for document filing, guidelines for completing trainee information, and policies and procedures outlining where and for how long trainee and assessor data should be kept

• How to conduct assessment, including the timing and duration specifics. Consideration may need to be given to operational issues where conducting assessment could interfere with day to day operations. A certain degree of flexibility is still needed in order for the training and assessment to remain relevant and valid

• Flexibility in order to respond to individual needs of participants who may have particular challenges such as physical or language and literacy needs. The assessment system must include features that cater for individual differences of participants including access and equity for all participants

• Outline country specific occupational safety and health regulations, employment requirements and equal opportunity legislation

• Procedures for recognising previous educational qualifications or experience gained at other training organisations or in other employment contexts. The assessment system should provide a mechanism that allows for prior learning and experience to be recognised if relevant to the current training or learning context

• Information about where assessment materials are located and where assessment can take place. This may be in a training room, work setting, computer lab, outside or a combination of places

• A review and evaluation process to determine if the assessment system is effective or needs improvement. This should be an ongoing process and should involve all stakeholders’ input

• Policies and procedures that allow for feedback including grievances, moderation, re-assessment

• A statement relating to competency levels and remuneration levels appropriate to those skills attained.


12



Introduction

One of the most important stages of an effective evaluation plan is to create a working group to conduct the evaluation. The members of this group should be relevant stakeholders who have a stake or vested interest in the evaluation findings and those who are the intended users who can benefit most from the evaluation. Stakeholders are much more likely to support and act on the evaluation findings if they are involved in the evaluation process from the beginning. Having representatives of all key stakeholders involved in the evaluation also ensures that the data collected, analysed and reported, actually meets the needs of the assessment system and the needs of the stakeholders themselves.

Determine stakeholders’ needs

Encouraging stakeholders to be involved in the evaluation process can create many benefits. The main stakeholders in this context will generally include the people who will use the evaluation results, support or maintain the assessment system, or who are affected by the assessment system activities or evaluation results. Stakeholders are a valuable resource for:

• Determining and prioritising key evaluation questions

• Trialling data collection methods and tools

• Facilitating data collection

• Implementing evaluation activities

• Increasing the credibility of analysis and interpretation of evaluation information

• Ensuring evaluation results are used.

In relation to stakeholders, the evaluation group may need to ask

• Who are the assessment system/organisation stakeholders?

• How can we work with all of the stakeholders?

• How are stakeholders’ roles outlined in the evaluation plan?

Who are the assessment system or organisation’s stakeholders?

Before it is possible to determine stakeholder needs, it is first necessary to identify who the stakeholders are and who to include in the evaluation plan or working group. Stakeholders, in this context, are the consumers or desired audience of the evaluation results. The stakeholders for the evaluation of an assessment system may include:

• Learners

• Trainers and assessors

• Supervisors, managers, and business owners




• Government agencies

• Co-workers

• Customers.

Communicating to stakeholders

It is important to explore stakeholder expectations at the beginning of the evaluation and come to a shared understanding of roles and responsibilities as well as the purposes of the evaluation. Some stakeholders will be represented on the evaluation working group and some will not. It is important to include a clear communication plan in your evaluation plan in order to meaningfully engage all appropriate stakeholders and increase participation and buy-in for the evaluation as well as use of final results.

Activity: List the appropriate role for each stakeholder relevant to the evaluation and how and when you might engage him or her in the evaluation. It is important to consider a stakeholder’s expertise, level of interest, and availability when developing the communication plan. If there are specific deadlines for information such as a stakeholder vote or funding opportunity, it is important to note those as well. Additional columns could be added for comments.

Evaluation Stakeholder

Role related to the Evaluation

Mode of Communication

Timing of Communication

A note on roles: Stakeholders are not necessarily members of the evaluation working group but may still have a role related to the evaluation. Given a stakeholder’s specific expertise, interest, availability, or intended use of the evaluation results, he or she may be involved in part or all of the evaluation without being a specific member of the working group. Roles might include but are not limited to:

• Development of the evaluation plan

• Feedback on focusing the evaluation

• Needing information about specific evaluation activities or progress of the evaluation

• Facilitating implementation of specific aspects of the evaluation

• Participation in interpretation meetings

• Disseminating and promoting use of evaluation results.


14


Stakeholder needs

While focusing the evaluation occurs later in the process, the groundwork begins with the identification of stakeholders relevant to the evaluation or the primary intended users. The evaluation working group members are designed to reflect the priority information needs of those members who will use the evaluation information. However, it is not always possible to include some high information need groups, and it is certainly not possible to include representation from every group that would benefit from evaluation results. This should not prevent evaluation staff and the working group from considering all points of view and needs for information when considering how best to focus the evaluation. Therefore, determining stakeholder information needs is both useful for considering the make-up of the working group and for focusing the evaluation.

Activity: From your list of primary intended users (those who have a stake in the evaluation results), identify what information each stakeholder will use.

Stakeholder Evaluation information needed

Some examples of stakeholder needs could include:

• Integrity and reliability of the outcomes of assessments

• Compliance with legislated and other obligations

• Cost-effectiveness of assessments

• Availability of, and ease of access to, assessment

• Availability of supplementary assessments to facilitate attainment of competency

• Record keeping requirements, such as detailing candidates and competencies achieved or attempted by candidates

• Feedback provided by assessors in relation to individual worker performance, attitude, capability, potential

• Specific workplace needs, such as improved profit, better productivity, enhanced safety, skilling of staff, improved products and services.





Introduction

The amount of information you can gather concerning the assessment system is potentially limitless. Evaluations, however, are always restricted by the number of questions that can be realistically asked and answered with quality, the methods that can be employed, the feasibility of data collection, and the available resources. These are the issues at the heart of this part of the framework: focusing the evaluation. The scope and depth of any system evaluation is dependent on assessment system and stakeholder priorities; available resources, including financial resources; staff and contractor availability; and amount of time committed to the evaluation.

What resources are needed?

Discussion of budget and resources (financial and human) that can be allocated to the evaluation will likely be included in your feasibility discussion. As a starting point, it is recommended that at least 10% of your total assessment system resources be allocated to review and conduct assessment evaluation. The questions and subsequent methods selected will have a direct relationship to the financial resources available, evaluation team member skills, and environmental constraints (e.g., you might like to do in-person interviews of assessors, but they may be external assessors who are located in another area). Stakeholder involvement may facilitate advocating for the resources needed to implement the evaluation necessary to answer priority questions. However, sometimes, you might not have the resources necessary to fund the evaluation questions you would like to answer most. A thorough discussion of feasibility and recognition of real constraints will facilitate a shared understanding of what the evaluation can and cannot deliver. The process of selecting the appropriate methods to answer the priority questions and discussing feasibility and efficiency is an ongoing process and will change according to resource constraints.

Brainstorming the possible resources that may be required to complete the evaluation will provide a clearer understanding of what is realistically achievable and what is not. Understanding stakeholder requirements will also help to determine what resources may be needed. In developing an idea of resources that may be needed, the following resource components should be considered:

• Financial resources, including money to fund staff involved in the evaluation, to purchase necessary materials, to pay for travel, to support data gathering, analysis and interpretation

• Physical resources, including equipment, materials, transportation, computer access, venue or office space

• Human resources, including internal staff and external personnel, as appropriate.


16


Useful evaluations are not about special research interests or what is easiest to implement but what information will be used by the system, stakeholders (including funders), and decision makers to improve the program and make decisions. Establishing the focus of the evaluation began with the identification of the primary purposes and the primary intended users of the evaluation. This process was further solidified through the selection of an evaluation working group. Developing the purposeful intention to use evaluation information and not just produce another evaluation report starts at the very beginning with assessment system planning and your evaluation plan. You need to garner stakeholder interests and prepare them for evaluation use. This step facilitates conceptualizing what the evaluation can and cannot deliver.

It is important to collaboratively focus the evaluation design with your working group based on the identified purposes, assessment system context, evaluation methods, and available resources. Additionally, issues of priority, feasibility, and efficiency need to be discussed with the working group and those responsible for the implementation of the evaluation. Transparency is particularly important in this step. Stakeholders and users of the evaluation will need to understand why some questions were identified as high priorities while others were rejected or delayed


Introduction

The assessment system assessors and staff should work together with the evaluation working group to determine the priority and feasibility of the evaluation questions and identify the uses of results before designing the evaluation plan. In this part of the plan, you will apply the purposes of the evaluation, its uses, and the assessment system description to narrow the evaluation questions and focus the evaluation for program improvement and decision making. In this step, you may begin to notice the iterative process of developing the evaluation plan as you revisit aspects of previous steps to inform decisions to be made later.

Developing the plan

An evaluation plan is a written document that describes how you will monitor and evaluate your assessment system, so that you will be able to describe the “What,” the “How,” and the “Why It Matters” for the system. The “What” reflects the description and accomplishments of the system. Your plan serves to clarify the assessment system’s purpose, anticipated expectations, and outcomes. The “How” answers the question, “How did you do it?” and assesses how the system is being implemented and if the system is operating with alignment to the system’s policies and procedures. Additionally, the “How” answers assessment system delivery corrections that should be made during implementation. The “Why It Matters” represents how the system makes a difference to the organisation as a whole. Being able to demonstrate that the assessment system has made a difference can be critical to its sustainability. An evaluation plan is similar to an assessment system work plan in that it is a roadmap and is used to guide the planning of activities used to assess the processes and outcomes of the assesment system. An effective evaluation plan is a dynamic tool that can change over time, as needed, to reflect system changes and priorities. An evaluation plan creates directions for accomplishing the system’s goals and objectives by linking evaluation and assessment system planning.




Ideally, assessment staff, evaluation staff, and the working group will be developing the evaluation plan while the assessment system is developing the work plan. Developing the evaluation plan simultaneously with the assessment work plan allows assessment system staff and stakeholders to realistically think through the process and resources needed for the evaluation. It facilitates the link between assessment planning and evaluation and ensures creating a feedback loop of evaluation information for assessment system improvement and decision making.

Ideally, the working group will develop one overarching evaluation plan that consolidates all activities and provides an integrated view of the assessment system.

However, your plan should be adapted to your specific evaluation needs and context. Additionally, it is important to remember that your evaluation plan is a living, dynamic document designed to adapt to the complexities of the environment within which the assessment system components are implemented. The plan is a guide to facilitate intentional decisions. If changes are made, they are documented and done intentionally with a fully informed working group and stakeholders.

Therefore, much care must go into the design of your data collection methods to assure accurate, credible and useful information. To really understand and assess an assessment system, multiple and mixed methods are required:

• “Multiple methods” means collecting data from more than one source and not relying on one survey or test or focus group to provide an adequate system assessment

• “Mixed methods” means that a variety of types of information sources are used to assess the assessment system.

Good evaluation usually combines both quantitative and qualitative methods. Quantitative methods gather numerical data that can be summarized through statistical procedures. Qualitative methods collect non-numerical data, usually textual, that can provide rich details about your project. Each approach has its particular strengths and, when used together, can provide a thorough picture of your project.

An example of an evaluation checklist can be found at:

http://www.wmich.edu/evalctr/wp-content/uploads/2010/05/plans_operations1.pdf

This evaluation checklist forms an important part of the evaluation plan and enables the working group to ensure that all aspects of the plan are accounted for.


18


Activity: Evaluation plan methods grid

Add some evaluation questions below and complete the table. Discuss answers with your partner.

Evaluation question

Indicator/ Performance Measure

Method Data Source Timeline Responsibility

How many candidates achieved competency?

Numbers of competent candidates

90% minimum set as standard

Data analysis Results documents

Observations

12 month period

Evaluation working group member A





Introduction

The evidence you gather to support the answers to your evaluation questions should be seen as credible by the primary users of the evaluation. The determination of what is credible is often context dependent and can vary across the assessment system and stakeholders. This determination is naturally tied to the evaluation design, implementation, and standards adhered to for data collection, analysis, and interpretation. Best practices for the assessment system and the evaluation standards which define what evidence is credible will facilitate a discussion with the evluation working group. This discussion allows for stakeholder input as to what methods are most appropriate given the questions and context of your evaluation. As with all the steps, transparency is important to the credibility discussion as well as the documentation of limitation of the evaluation methods or design. This facilitates the likelihood that results will be more acceptable to stakeholders and strengthens the value of the evaluation and likelihood the information will be used for system improvement and decision making. The value of stakeholder inclusion throughout the development of the evaluation plan is prominent in the recognition of evidence gathering credibility once the plan has been completed.

What evidence should be collected?

Gathering the appropriate evidence for the evaluation is a critical aspect of the task. Collecting data or information that is not relevant to the evaluation questions or goals is ineffective and could waste time and money. Once the evaluation questions have been developed, stakeholder needs and expectations are identified, and an evaluation working group has been established, potential sources of evidence need to be identified and then checked to ensure that the data and information aligns with the evaluation purpose, goals and questions.

Some examples of evidence that needs to be captured may include:

• Rolls and other attendance forms that record participation in assessment

• Samples of assessment evidence used by assessors to determine competency

• Reports relating to assessment-related meetings held by trainers, assessors and learners/candidates

• Information provided to learners regarding their assessment

• Sample assessment items, such as tests, requirements for demonstrations, third party reports, observation checklists, portfolios of work

• Criteria used for judging competency

• Proof of expenditure relating to the cost of assessments

• Evidence of learner progress.


20



Introduction

Gathering relevant evidence for the evaluation is a critical factor in ensuring the quality of the outcomes. It is also important to determine how the effectiveness of the assessment system will be judged. What criteria will be used to assess the overall success or failure of the system? When defining the criteria for assessing the effectiveness of the system, it is important to make sure that the criteria aligns with the purpose and goals of the evaluation. The criteria must match and provide answers to the evaluation questions.

What criteria should be used?

As emphasized already, it is important to select the method(s) most appropriate to answer the evaluation questions. The types of data needed should be reviewed and considered for credibility and feasibility. Based on the methods chosen, you may need a variety of input, such as case studies, interviews, assessment material examples, focus groups, standardized indicators, and surveys. You may need to consider multiple data sources and the triangulation of data for reliability and validity of your information. Data may come from external sources or agencies or gathered from system-specific sources (either existing or new).The form of the data (either quantitative or qualitative) and specifics of how these data will be collected must be defined, agreed upon as credible, and transparent. There are strengths and limitations to any approach, and they should be considered carefully with the help of the working group. For example, the use of existing data sources may help reduce costs, maximize the use of existing information, and facilitate comparability with other assessment systems, but may not provide assessment system specificity to the sytem you are evaluating. Additionally, existing sources of data may not meet the question-method appropriateness criteria. In determining the appropriate criteria to be used for the evaluation, it is important to check:

• The evaluation questions

• The stakeholder needs and desired outcomes

• The purpose of the evaluation.

Evaluative criteria are the standards, measures, or expectations about what should exist against which measures of actual performance are compared and evaluated. Evaluators should select evaluative criteria that are relevant, appropriate and sufficient to address the evaluation’s objectives. Unlike financial or performance audits, the objectives of system evaluations generally are not to assess a system’s or agency’s compliance with legal requirements (although this may be a part of the evaluation) but to assess whether the system’s expectations have been met. The sources of those expectations can be quite diverse. However, if the intended audience for the report —both the study requesters and system managers — believes that the chosen criteria and measures are appropriate, and then the study’s findings are more likely to be credible.

Depending on the circumstances of the program and the evaluation questions, examples of possible criteria include:

• Purpose or goals prescribed by law or regulation




• Policies or procedures established by internal/external officials

• Professional standards or norms

• Expert opinions

• Prior period’s performance

• Performance of other entities or sectors used to benchmark performance.

Some criteria designate a particular level as distinguishing acceptable from unacceptable performance, such as in determinations of assessor qualifications and experience. Related evaluation questions ask whether a system’s performance is “acceptable” or “meets expectations.” Other criteria have no pre-established level designated as representing acceptable performance but permit assessment of the extent to which expectations are met. Thus, while the evaluation cannot typically ascertain whether an assessment system was “effective” per se, it can compare the performance of a system across time and to the performance of other systems or organizations to ascertain whether it is more or less effective than other efforts to achieve a given objective.

To support objective assessment, criteria must be observable and measurable events, actions, or characteristics that provide evidence that performance objectives have been met. Some legislation, evaluation requests, or program designs provide broad concepts for performance objectives, such as “a thorough process” or “participant well-being,” that lack clear assessment criteria. In such cases, the evaluator may need to gain the agreement of study requesters and system managers to base assessment criteria on measures and standards in the subject matter literature.

Revisiting aspects of the evaluation will enable a clearer idea of what criteria will be the most effective and relevant to the evaluation. Some of the criteria that may be useful in determining the effectiveness of the assessment system may include:

• Ratio of learners attempting assessment to the numbers who achieve competency

• Numbers of candidates who have achieved competency

• Ratio of candidates who have re-tested prior to achieving competency

• Alignment of assessment activities with curriculum and competency requirements

• Internal statistics relating to expectations regarding achievement of competency

• Cost

• Workplace benefits achieved as a result of competency being achieved

• The amount of time learners spend involved in assessment, including preparation for assessment. This should address workplace time and time contributed out of hours by candidates and address ancillary issues such as waiting time, travelling.

Activity: In pairs or small groups, list any other criteria that may help determine the effectiveness of the assessment system.


22



Introduction

It is important to remember that not all methods fit all evaluation questions and often a mixed-methods approach is the best option for a comprehensive answer to a particular evaluation question. This is often where you need to consult with your evaluation experts for direction on matching method to question. Note that all data collected needs to have a clear link to the associated evaluation question and anticipated use to reduce unnecessary burden on the respondent and stakeholders. It is important to revisit data collection efforts over the course of an evaluation plan to check effectiveness and ensure that the methods are appropriate. Finally, it is not enough to have defined measures. Quality assurance procedures must be put into place so that data is collected in a reliable way, coded and entered correctly, and checked for accuracy. A quality assurance plan should be included in your evaluation plan.

Qualitative versus Quantitative

Often, when involved in data gathering, a debate between qualitative and quantitative methods usually arises. It is not that one method is right and one method is wrong, but which method or combination of methods will obtain answers to the evaluation questions.

Some options that may point you in the direction of qualitative methods:

• You are planning and want to assess what to consider when designing an assessment system or initiative. You want to identify elements that are likely to be effective

• You are looking for feedback while a system or initiative is in its early stages and want to implement a process evaluation. You want to understand approaches to enhance the likelihood that an initiative (e.g. policy or procedural change) will be adopted

• Something isn’t working as expected and you need to know why. You need to understand the facilitators and barriers to implementation of a particular initiative

• You want to truly understand how a system is implemented on the ground and need to develop a model or theory of the system or initiative.

Some options that maypoint you in the direction of quantitative methods:

• You are looking to identify current and future movement or trends of a particular phenomenon or initiative

• You want to consider standardized outcome across assessment as a whole. You need to monitor outputs and outcomes of an initiative. You want to document the impact of a particular initiative

• You want to know the costs associated with the implementation of a particular aspect of the assessment system

• You want to understand what standardized outcomes are connected with a particular initiative and need to develop a model or theory of the program or initiative.




Or the most appropriate method may be a mixed methods approach wherein the qualitative data provide value, understanding, and application to the quantitative data. The question is not whether to apply qualitative or quantitative methods but what method is most appropriate to answer the evaluation question chosen.

To choose the appropriate methods in your evaluation plan, you will need to:

• Keep in mind the purpose, assessment system description, stage of development of the system, evaluation questions, and what the evaluation can and cannot deliver

• Confirm that the method(s) fits the question(s); there are a multitude of options, including but not limited to qualitative, quantitative, mixed-methods, multiple methods

• Think about what will constitute credible evidence for stakeholders or users

• Identify sources of evidence (e.g. persons, documents, observations, administrative databases, surveillance systems) and appropriate methods for obtaining quality (i.e. reliable and valid) data

• Identify roles and responsibilities along with timelines to ensure the project remains on-time and on-track

• Remain flexible and adaptive, and as always, transparent

• Consider the cost of various methods.

What are your evaluation questions?

If you are trying to learn...

How many?

How much?

What percentage?

How often?

What is the average amount?

If you are trying to learn...

What worked best?

What did not work well...?

What do the numbers mean?

How was the project useful...?

What factors influenced success or failure?

Choose quantitative methods Choose qualitative methods


24


Different methods

There are various methods that can be used for evaluation. These could include:

• Matching actions and models to identified need

• Interviews with group and/or one-on-one

• Surveys

• Questionnaires

• Various models which have been designed by researches including: stake model, stufflebeam model, Scriven model, ethnographic model, action research model, the Portrayal model.

Activity: Visit the various links below and write a short description of each model. Which ones are the best for your organisation’s needs? Why?

Evaluation Model Overview:

http://www.google.com.vn/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CD0QFjAC&url=http%3A%2F%2Ftalc.ukzn.ac.za%2FLibraries%2FCurriculum%2Fmodels_of_curriculum_evaluation.sflb.ashx&ei=UnLjUYjQMKW9iAfUjYCADA&usg=AFQjCNGzYB0TDw1YWcvXBN_p6_2djbeurg&bvm=bv.48705608,d.aGc

Stufflebeam:

http://www.aseesa-edu.co.za/bullh.htm

Evaluation Checklists:

http://www.wmich.edu/evalctr/wp-content/uploads/2010/05/plans_operations1.pdf





Introduction

Once evaluation questions have been formulated, the next step is to develop an evaluation design - to select appropriate measures and comparisons that will permit drawing valid conclusions on those questions. In the design process, the evaluator explores the variety of options available for collecting and analyzing information and chooses alternatives that will best address the evaluation objectives within available resources. Selecting an appropriate and feasible design, however, is an iterative process and may result in the need to revise the evaluation questions.

Key components of evaluation design

An evaluation design documents the activities best able to provide credible evidence on the evaluation questions within the time and resources available and the logical basis for drawing strong conclusions on those questions. The basic components of an evaluation design include the following:

• The evaluation questions, objectives, and scope

• Information sources and measures, or what information is needed

• Data collection methods, including any sampling procedures, or how information or evidence will be obtained

• An analysis plan, including evaluative criteria or comparisons, or how or on what basis program performance will be judged or evaluated

• An assessment of study limitations.

Clearly articulating the evaluation design and its rationale in advance aids in discussing these choices with the requester and other stakeholders. Documenting the study’s decisions and assumptions helps manage the study and assists report writing and interpreting results.

The design phase of an evaluation is a period for examining options for answering the evaluation questions and for considering which options offer the strongest approach, given the time and resources available. After reviewing materials about the system, evaluators should develop and compare alternative designs and assess their strengths and weaknesses. For example, in choosing between using system administrative data or conducting a new survey of assessment staff, the evaluator might consider whether 1) the new information collected through a survey would justify the extra effort required, or 2) a high quality survey can be conducted in the time available.


26


Depending on the assessment system and study question, potential sources for evidence on the evaluation question include system administrative records, assessor reports, performance monitoring data, surveys of system participants, and existing surveys of external agencies or private or public facilities. In addition, the evaluator may choose to conduct independent observations or interviews with external officials, system participants, or persons or organizations doing business with the organisation.

In selecting sources of evidence to answer the evaluation question, the evaluator must assess whether these sources will provide evidence that is both sufficient and appropriate to support findings and conclusions on the evaluation question. Sufficiency refers to the quantity of evidence — whether it is enough to persuade a knowledgeable person that the findings are reasonable. Appropriateness refers to the relevance, validity, and reliability of the evidence in supporting the evaluation objectives. The level of effort required to ensure that computer-processed data (such as participant records) are sufficiently reliable for use will depend on the extent to which the data will be used to support findings and conclusions and the level of risk or sensitivity associated with the evaluation.

Measures are the concrete, observable events or conditions (or units of evidence) that represent the aspects of assessment system performance of interest. Some evaluation questions may specify objective, quantifiable measures, such as the number of participants, who gained competency or qualitative measures, or qualitative such as the reasons for noncompliance. But often the evaluator will need to select measures to represent a broader characteristic, such as “service quality.” It is important to select measures that clearly represent or are related to the performance they are trying to assess. For example, a measure of the average processing time for tax returns does not represent, and is not clearly related to, the goal of increasing the accuracy of tax return processing. Measures are most usefully selected in concert with the criteria that the system performance will be assessed against, so that agreement can be reached on the sufficiency and appropriateness of the evidence for drawing conclusions on those criteria.

Additional considerations for assessing the appropriateness of existing databases include: whether certain subgroups of the population are well-represented; whether converting data from its original format will require excessive time or effort; and when examining multiple sites, whether variation in data across sites precludes making reliable comparisons. No data source is perfectly accurate and reliable; thus, evaluators often consider using multiple measures or sources of data to triangulate toward the truth. Concerns about biases in one data source — for example, possible exaggerations in self reports of employment history — might be countered by complementing that information with similar measures from another source —for example, length of employment recorded in administrative records.

Tools to capture identified evidence

There are many tools that could be developed and utilised to gather relevant evidence for the evaluation and these could include:

• Survey instruments such as questionnaires, diaries, logs, attitude scales, diagnostics

• Interview schedules and records

• Observation sheets and checklists

• Objective product analysis sheets




• Identified evidence such as:

� Affective evidence, i.e. Satisfaction with the program

� Cognitive evidence, i.e. Relating to knowledge and/or skills gained

� Performance or behaviour, i.e. Relating to quality of work performed after training/assessment, productivity.

No matter how data are collected, care should be taken to ensure that data are sufficient and appropriate to support findings on the evaluation question. Trained observers may inspect physical conditions, actions or records to ascertain whether these met requirements or other kinds of criteria. When collecting testimonial evidence through interviews or surveys, the evaluator should consider whether the people serving as data sources are sufficiently knowledgeable and whether their reports of events or their opinions are likely to be candid and accurate. In addition, careful attention to developing and pretesting questionnaire surveys and other data collection instruments will help ensure that the data obtained are sufficiently accurate for the purposes of the study. Where the evaluator aims to aggregate and generalize from the results of a sample survey, great importance is attached to collecting uniform data from every unit in the sample. Consequently, sample survey information is usually acquired through structured interviews or self-administered questionnaires. Most of the information is collected in close-ended form, which means that the respondent chooses from responses offered in the questionnaire or by the interviewer. Designing a consistent set of responses into the data collection process helps establish the uniformity of data across units in the sample.


28


Work Projects

It is a requirement of this Unit that you complete Work Projects as advised by your Trainer. You must submit documentation, suitable evidence or other relevant proof of completion of the project to your Trainer by the agreed date.

1.1 Write a purpose statement for an evaluation of the assessment system at your workplace. Submit to your trainer.

1.2. Provide your trainer with an outline of the different components of an assessment system and put them into categories:

• Administration, training, materials, policies etc.

1.3. Provide a list of all the relevant stakeholders in your organisation and determine what their needs are in relation to the assessment system.

1.4. Develop an outline of the resources that will be needed to conduct an evaluation plan.

1.5 Develop a draft evaluation plan that:

• Outlines the purpose of the plan

• Determines the evidence that needs to be gathered

• Defines the criteria for determining effectiveness of the system.

1.6 Provide a list of different methods for capturing and analysing data and explain the main features of each one to your trainer.

1.7 Develop a questionnaire, interview or survey that responds to evaluation questions within your plan. This could relate to one part of the evaluation only.




Summary

Plan the evaluation of the assessment system

Developing an evaluation plan for an assessment system includes many different factors. Good planning helps to maintain the credibility and effectiveness of the evaluation and helps to identify any challenges before beginning the evaluation itself. In developing an effective evaluation plan, you should consider:

• The purpose and role of the evaluation

• Outlining all of the parts of the assessment system before developing the plan

• Checking carefully what resources, both human and materials, may be required to complete the evaluation

• The needs of all stakeholders and what outcomes from the evaluation you/they are seeking

• Developing a comprehensive evaluation plan that includes methods, evaluation questions, criteria for judging the effectiveness of the system, and tools to capture relevant and useful evidence of the assessment system’s effectiveness.


30







Introduction

When engaging in primary (or original) data collection, it is important to conduct a pretest or pilot study before beginning full-scale data collection. The pilot study gives the evaluator an opportunity to refine the design and test the availability, reliability, and appropriateness of proposed data. Evaluators new to the system or proposing new data collection may find that a limited exploration of the proposed design in a few sites can provide a useful “reality check” on whether one’s assumptions hold true. The pilot phase allows for a check on whether system operations, such as participant recruitment, and delivery of services occur as expected. Finding that they do not may suggest a need to refocus the evaluation question to ask why the system has been implemented so differently from what was proposed. Testing the work at one or more sites allows the evaluator to confirm that data are available, the form they take, and the means for gathering them, including interview procedures. It also provides an opportunity to assess whether the analysis methodology will be appropriate.

Trial data tools

Once the data gathering tools have been developed, it is important to test their effectiveness and relevance to the evaluation purpose. If the questionnaires, surveys, or interview information gathered does not reflect the information required to answer the evaluation questions, then they need to be altered in order to capture useful information. Choosing some sample areas to gather data helps to direct the evaluation plan in the right direction. It would not be effective to conduct the entire data gathering process and then realise that the data captured is not enough or irrelevant to the evaluation questions.

After gathering some sample results from different aspects of the evaluation, it is necessary to ask some questions that relate to the data, such as:

• Was the process easy to implement?

• Did all participants understand what they had to do?

• Is the data relevant to the evaluation questions and purpose?

• Were the techniques cost effective?

• Are there enough people available to conduct the evaluation?

• What issues need to be rectified?

• Is the reporting mechanism effective?

• Can the data be easily stored and retrieved?


32


Once the trial has been completed it is important to document any changes, updates, issues, or suggestions from the pilot program in order to help increase the validity and credibility of the data gathering process. A checklist could be useful for this. (See below)

Data gathering tool When, what and who conducted?

Issues/Concerns Suggested changes

Assessor qualification records

12/6/2012

Bob Jones

7 samples of qualification records were observed and checked with individual assessors

It was difficult to schedule meetings with assessors.

Negotiate a timetable for meetings with assessors well in advance of conducting interviews.





Introduction

After gathering samples of data and trialing the data gathering tools and techniques, and issues or problems need to be identified and changed to ensure the overall quality and integrity of data. These suggestions and changes need to be documented and agreed upon by stakeholders in order for the credibility of the data to be acknowledged once the evaluation has been completed.

Revising data gathering tools and techniques

After trialing different aspects of the data gathering tools and techniques, the data collected needs to be checked for relevance and effectiveness. In determining the effectiveness of the trial data collected, the process and the content need to be assessed. The content or data, for example, may be useful and appropriate, but the process of gathering the data may be costly, time consuming, or difficult to arrange. In contrast, the process may be easy and cost effective but the data obtained may be irrelevant or there may be too much information to analyse. Revision to the data gathering tools and techniques would then need to be made.

Some issues that may arise from the pilot process could include:

• Challenges in gathering the data: time, location, cost, compiling the data, not enough human resources, not enough data records available

• Issues with the content: questionnaire questions, survey questions, too many questions or not enough, language challenges for understanding questions,

• Data doesn’t reflect evaluation questions or purpose

Activity: Complete the table below and make suggestions on how to revise the tools to meet the needs of stakeholders.

Data gathering tool/technique

Trial data gathered Finding/Issues Revisions needed

Interviews with assessors re: assessment system procedure knowledge

10 interviews conducted and interview questions asked and responses recorded on video

Responses and questions appropriate

Transcribing responses extremely time consuming

Create a paper-based or online knowledge test for assessors to complete

Questionnaire for assessment system administration staff

Questionnaire delivered and completed by 12 admin staff

Too many written responses

Hand writing illegible

Too much data


34


Data gathering tool/technique

Trial data gathered Finding/Issues Revisions needed

Assessment records of training participants

22 assessment records checked from prvious 12 months

Storage of records in paper form

Filing system difficult to understand

Limited staff available to find records

External training provider interviews and site visits

2 training provider facilities visited and trainer/staff interviews conducted

Training providers were reluctant to meet with evaluation team

Would not provide access to records or data

Focus group with assessors

10 assessors and managers and coordinators invited

4 particpants turned up to focus group

No replies to email invitation received





Introduction

When the trial of the data gathering tools and techniques has been completed, data reviewed, and revisions made, the evaluation plan evidence collection process can begin. There may be a lot of people involved in this process and there may also be different locations and times for data collecting activities. There will also be a timeline or schedule for completing activities so it is important to manage this part of the evaluation effectively.

Planning the data collection activities

There are a lot of components and different activities related to the data collection for the evaluation plan and it is also one of the most important parts of the plan. Collecting relevant, sufficient and credible evidence will be the basis for how the overall success or failure of the evaluation process is judged. In order to manage this process effectively it would be useful to create a process management plan or template to control and manage the data collection activities.

Gantt Chart

Below is an example of a Gantt Chart which may be a useful activity management tool. It enables the project manager the ability to check who is doing what and when. It also helps to provide an update to stakeholders on the overall progress of the evidence gathering activities.

Evaluation – Data Collection

Unique ID WBS Name Responsible

Start Date

Finish Date

% Complete

1 1 Arrange data collection resources

Anne 3/19/2010 4/1/2010 75%

2 1.1 - Assign evidence gatherers/staff

Anne 3/19/2010 4/1/2010 100%

3 1.2 - Distribute resource materials

Anne 3/19/2010 4/1/2010 25%

4 2 Obtain Evidence Gathering Approval

Brett 3/31/2010 4/9/2010 50%

5 3 Collect data and evidence Brett 5/28/2010 7/16/2010 0%

6 3.1 - Arrange transport for offsite activities

Brett 5/28/2010 6/18/2010 70%

7 3.2 - Arrange assessor interviews and conduct

Anne 5/28/2010 6/25/2010 20%

8 3.3 - Gather assessment record data

Dave 5/28/2010 6/25/2010 15%

9 4 Collate and store data Sue 4/5/2010 6/25/2010 10%


36


Collecting data

When collecting the identified evidence it is important to:

• Use the data gathering tools and techniques as planned

• Make sure the evidence is sufficient

• Ensure privacy and confidentiality

• Check the accuracy of the data

• Allow for changes in data collection if there is not enough or inappropriate data gathered

• Communicate with stakeholders

• Ignore irrelevant data

• Follow-up on relevant issues that are identified as part of the evaluation process

• Make sure that all relevant data is gathered.

Activity: List all of the main tasks for collecting evidence and under each task list the sub-tasks that will have to be completed. Make sure that they follow a logical order. Share with you partner or group and identify any tasks that you may have forgotten.




2.4 Store data

Introduction

There will be a lot of information, data and evidence collected during the evaluation and it is critical that you are able to provide easy access to this material in order to be able to analyse it effectively. There are many things to consider when storing data so that it can be retrieved easily. It is important to consider cost constraints, facilities and staff resources available to organise and maintain data storage.

Storing data effectively

In the past all records and data would have to be kept in a paper-based form but with the development of new technology, such as computers and database files, vast amounts of information can be easily stored and accessed. A lot of the information from interviews and surveys might still be paper-based and will have to be filed or stored in a way that is easy to find and access. The paper-based data may be transferred to computer files but there may be regulations that stipulate the storage of all documents for a certain amount of time. Storage of paper-based documents and files will require space and a filing system.

The data that has been collected will need to be classified into different groups of information and each section of information will then have to be further classified. In the case of the evaluation, the data could be classified under assessment toll types such as questionnaires, surveys, interviews and records. Alternatively, the data could be stored in relation to the evaluation questions. Either way will require a document that explains how the information has been stored.

Choosing a logical and consistent way to name and organise your files allows you and others to locate and use them. Ideally, the best time to think how to name and structure the documents and directories you create is at the start of a project.

Agreeing on a naming convention will help to provide consistency, which will make it easier to find and correctly identify your files, prevent version control problems when working on files collaboratively, and generally prevent errors in the evaluation. organising your files carefully will save you time and frustration and prevent duplication or errors by helping you and your colleagues find what you need when you need it.

Whether you are working on a stand alone computer, or on a networked drive, the need to establish a system that allows you to access the files, avoid duplication and that can be backed up, takes a little planning. A good place to start is to develop a logical folder structure. The following tips should help you develop such a system:

Use folders – group files within folders so information on a particular topic is located in one place.

Adhere to existing procedures – check for established approaches in your team or department which you can adopt.

Name folders appropriately – name folders after the areas of data to which they relate to. This avoids confusion in shared workspaces if a member of staff leaves, and makes the file system easier for new staff or subsequent projects to navigate.


38


Be consistent – When developing a naming scheme for your folders it’s important that once you’ve decided on a method, you stick to it. If you can, try to agree on a naming scheme from the outset of your research project.

Structure folders hierarchically – start with a limited number of folders for the broader topics, and then create more specific folders within these (sub-folders).

Transferring paper-based data to electronic – if you intend to transfer paper-based data to electronic versions, it is critical that the data is transferred correctly. It is important to check that the data is the same as the paper-based data.

Backup – Ensure that your files, whether they are on your local drive or on a network drive, are backed up. Paper-based records will provide back-up to electronic versions.

Review records – Assess materials regularly or at the end of a project to ensure files aren’t kept needlessly. Put a reminder in your calendar so you don't forget!

Security and confidentiality – It is important to maintain confidentiality and security of data and this may be an external or internal requirement. It is necessary to identify who and how access is granted to the data and to develop procedures for data use.

Decide on a file naming convention at the start of your project. Useful file names are:

• Consistent

• Meaningful to you and your colleagues

• Allow you to find the file easily.

It is useful if your stakeholders agree on the following elements of a file name:

Vocabulary – choose a standard vocabulary for file names, so that everyone uses a common language.

Punctuation – decide on conventions for if and when to use punctuation symbols, capitals, hyphens and spaces.

Dates – agree on a logical use of dates so that they display chronologically i.e. YYYY-MM-DD.

Order - confirm which element should go first, so that files on the same theme are listed together and can therefore be found easily.

Numbers – specify the amount of digits that will be used in numbering so that files are listed numerically e.g. 01, 002, etc.





Introduction

Justifying conclusions includes analyzing the information you collect, interpreting, and drawing conclusions from your data. This step is needed to turn the data collected into meaningful, useful, and accessible information. This is often when programs incorrectly assume they no longer need the evaluation working group integrally involved in decision making and instead look to the “experts” to complete the analyses and interpretation. However, engaging the working group in this step is critical to ensuring the meaningfulness, credibility, and acceptance of evaluation findings and conclusions. Actively meeting with stakeholders and discussing preliminary findings

helps to guide the interpretation phase. In fact, stakeholders often have novel insights or perspectives to guide interpretation that evaluation staff may not have, leading to more thoughtful conclusions.

Analyse the data

Errors or omissions in planning this step can create serious delays in the final evaluation report and may result in missed opportunities if the report has been timed to correspond with significant events. Often, groups fail to appreciate the resources, time, and expertise required to clean and analyze data. This applies to both qualitative and quantitative data. Some evaluations focus their efforts on collecting data, but never fully appreciate the time it takes to work with the data to prepare for analysis, interpretation, feedback, and conclusions. These programs are suffering from “D.R.I.P.”, that is, programs that are “Data Rich but Information Poor.” Survey data remains “in boxes” or interviews are never fully explored for theme identification.

After planning for the analysis of the data, you have to prepare to examine the results to determine what the data actually say about the assessment system. These results should be interpreted with the goals of the system in mind, the context of the system, and the needs of the stakeholders.

Moreover, it is critical that your plans include time for interpretation and review by stakeholders to increase transparency and validity of your process and conclusions. The emphasis here is on justifying conclusions, not just analyzing data. This is a step that deserves due diligence in the planning process. The identifed standards and procedures play a role in guiding the evaluator’s decisions on how to analyze and interpret data to assure that all stakeholder values are respected in the process of drawing conclusions. That is to say, who needs to be involved in the evaluation for it to be ethical. This may include one or more stakeholder interpretation meetings to review interim data and further refine conclusions.

It is the responsibility of the evaluator and the evaluation working group to ensure that conclusions are drawn directly from the evidence. This is a topic that should be discussed with the working group in the planning stages along with reliability and validity issues and possible sources of biases. If possible and appropriate, triangulation of data should be considered and remedies to threats to the credibility of the data should be addressed as early as possible.


40


Analysing quantitative data

The first step in analyzing quantitative data is to summarize the responses using descriptive statistics. When you collect and summarize quantitative data, your result is a distribution of scores for each item on your survey (except open-ended items). A distribution is simply the collection of all ratings or scores for a particular item, ordered from the lowest to the highest value. The table below presents some of the most common descriptive statistics: frequency counts, percentages, and measures of central tendency (mean, median, and mode).

Question: Please indicate your level of agreement with this statement.

I know where to submit and store assessment records and materials.

Response Stronglyagree

Somewhat agree

Uncertain Somewhat disagree

Stronglydisagree

Total Missing

Response value (5) (4) (3) (2) (1)

N 100

Frequencies 54 36 5 2 0 97 3

Percent 54.0% 36.0% 5.0% 2.0% 0.0% 97.0% 3.0%

Valid percent 55.7% 37.1% 5.2% 2.1% 0.0%

Mean 4.41

Median 5

Mode 5

N Number of people responding to the survey. (Note: 100 people returned a survey, but only 97 responded to this particular question.)

Frequencies The number of respondents choosing each response.

Percent The number of those choosing that response divided by the number of people who completed the survey.

Valid percent

The number of respondents choosing that response divided by the number of respondents who answered the question. In this example, we had 100 people complete the survey, but only 97 actually responded to this particular question.

Meam The mean is the “average” response in your distribution. It is computed by adding all responses and dividing by the number of respondents who answered the question.

Median The median is the score that is in the middle of the distribution, with half of the scores above and half below. To find it, sort your distribution from highest to lowest ratings, then find the number that equally divides the distribution in half. For the 97 people who completed this distribution, the 49th score divides the distribution in half. The 49th (median) score is a “5.” When the majority of ratings fall either at the high or low end of a rating scale, as they do here, the median is usually the preferable measure of central tendency because it is not affected by a few extremely low or high ratings

Mode The mode is the most frequent response. For many demographic and two-option questions, the mode is the only measure of central tendency that can be reported. This is also true for questions that ask respondent to provide more than one response, such as “check all that apply” questions




Combined response table N = 50

Positive (Strongly

agree/Agree) Neutral (Neither

agree or disagree)

Negative (Disagree/Strongly

disagree)

The training session helped me develop more confidence in recording assessment results

39 78%

9 18%

2 4%

The training session helped me develop more confidence in using assessment procedures

32 64%

13 26%

5 10%

Analysis: This table makes the pattern of positive ratings more obvious for the items by combining positive, neutral and negative results. It also confirms that ratings were more positive for the recording assessment results session compared to the assessment procedures session. One explanation might be that assessment procedures session has a lot of information and requires a longer training session or more training sessions compared to recording assessment results session.

Tables are very helpful for understanding your data. The tables above show formats that will help you analyze your descriptive data. After you compile a table, write a few notes interpreting the numbers.You may simplify your data to make the positive and negative trends more obvious. For instance,

In the table above the “Strongly Agree” and “Agree” responses were combined into a “Positive” category and the “Disagree/Strongly Disagree” responses were put into a “Negative” category.

Checking the validity of data

Validity refers to the accuracy of the data collected through your survey: did the survey collect the information it was designed to collect? It is the responsibility of the evaluator to assess the factors that may affect the accuracy of the data and present those factors along with results. Threats to validity of surveys usually fall in one of the following categories:

• Response rate. When small percentages of respondents return surveys, the potential for bias must be acknowledged. Even when using the strategies discussed earlier, you may not obtain an adequate response rate. If resources allow, you can assess the degree of bias somewhat with follow-up interviewing or surveying of nonrespondents. For instance, if you suspect that those who responded were biased in the favourable direction, you could conduct a phone survey with a random selection of 10% of your respondents with a few simple questions to explore the extent of bias

• Low completion rate of specific sections of surveys. If many respondents do not complete certain sections of the survey, you will have to question the findings of that part of the survey. For instance, respondents may not finish the survey, leaving final sections or pages blank. To avoid this problem, keep your surveys as short as possible. For electronic surveys, provide a “progress bar” that tracks the percentage of questions completed as the respondent proceeds through the survey


42


• Low completion rate of questions. Even if you have a respectable response rate, you may have questions that are left blank by a number of respondents. There are several reasons why respondents do not answer particular questions. They may not find a response that applies to them, the question format may be confusing, or they do not understand the question. The best strategy for avoiding this problem is to carefully pilot your questions. If your survey asks questions that are sensitive or threatening, your best strategy for getting responses is to conduct an anonymous survey

• Socially desirable responding. Sometimes respondents are embarrassed to answer questions truthfully. If possible, avoid using questions that ask people to disclose information that may be embarrassing or threatening. This challenge may occur if your survey asks respondents to report personal information. If you must ask such questions, providing anonymity may enhance the accuracy of responses.

You cannot prove validity. You must build your case for the credibility of your survey by showing that you used good design principles and administered the survey appropriately. After data collection, you assess the shortcomings of your survey and candidly report how they may impact interpretation of the data.

Surveys allow you to collect a large amount of quantitative data, which then can be summarized quickly using descriptive statistics. This approach can give you a sense of the experience of participants in your evaluation and can allow you to assess how closely you have come to attaining your goals. However, based on the analysis given, you may find that the conclusions are tentative. This is because the numbers may describe what the respondents believe or feel about the questions you asked but they do not explain why participants believe or feel that way. Even if you include open-ended questions on your survey, only a small percentage of people are likely to take the time to comment.

For evaluation, the explanations behind the numbers usually are very important, especially if you are going to make changes to your assessment system or make decisions about canceling or continuing your efforts. That is why most evaluation plans include a combination of qualitative and quantitative methods.




Work Projects


2.1 Provide your trainer with a sample of a data gathering tool you have developed which you have trialled on some participants:

• Include a report on how effective or ineffective it was and any revision that needs to be made as a result of the trial.

2.2 Use one part of the evaluation plan and collect data relevant to that part:

• Record the data that you gathered in an appropriate form

• Provide an analysis of the results to your trainer.


44


Summary

Undertake the evaluation of the assessment system

Once a plan has been developed it is necessary to collect the evidence to determine the effectiveness of the assessment system. It is important to check the tools that will be used first before engaging in the full evaluation plan to determine if they are adequate and perform the role they are meant to. If after trialing the data gathering tools you find that there are some issues with the tools, it is critical that you adjust the tools to align them with the evaluation needs. Once the tools have been checked, you can:

• Collect the data that is required making sure the data is aligned to the overall goals and evaluation questions

• Store the data in an appropriate format once collected that makes the data easily accessible and keeps the information secure and confidential

• Analyse the results to determine the effectiveness of the assessment system.






Introduction

Evaluation reports are a useful way to describe system successes, as well as areas in need of improvement. When data is communicated well, it can be used to maintain current support, garner new backing, and apply for additional funding. The information in the report should be clear and understandable to those not directly involved in the evaluation plan implementation.

Evaluation report

The basic elements of an evaluation plan include:

• Title page

• Question overview

• Intended use and users

• Program description

• Evaluation focus

• Methods

• Analysis and interpretation plan

• Use, distribution of results, and recommendations.

Title page: This page provides easily identifiable name, dates covered, and possibly basic focus of the evaluation.

Question overview: In an evaluation plan, this is an overview of the evaluation questions for ease of reference, similar to the executive summary in a final evaluation report.

Intended use and users: This section fosters transparency about the purposes of the evaluation and who will have access to evaluation results. It is important to build a market for evaluation results from the beginning. This section identifies the primary intended users and the evaluation working group and describes the purposes and intended uses of the evaluation.

Assessment system description: This section provides a shared understanding of the description of system and a basis for the evaluation questions and prioritization. This section will usually include a narrative description. This section can also facilitate completing the introduction section for a final report or publication from the evaluation. This section might also include a reference section or bibliography related to system description.


46


Evaluation focus: There are never enough resources or time to answer every evaluation question. Prioritization must be collaboratively accomplished based on the assessment system description, the stage of development of the system, assessment and stakeholder priorities, intended uses of the evaluation, and feasibility issues. This section will clearly delineate the criteria for evaluation prioritization and will include a discussion of feasibility and efficiency. It will also outline who will be responsible for what and when the actions will be completed. There may also be a section that identifies any training needs of staff who will be involved in evaluating the system.

Methods: This section covers indicators and performance measures, data sources and selection of appropriate methods, roles and responsibilities, and credibility of evaluation information. This section will include a discussion about appropriate methods to fit the evaluation question. An evaluation plan methods grid is a useful tool for transparency and planning.

Analysis and interpretation plan: Who will get to see interim results? Will there be a stakeholder interpretation meeting or meetings? It is critical that the plan allows time for interpretation and review from stakeholders (including your critics) to increase transparency and validity of the process and conclusions. The emphasis here is on justifying conclusions, not just analyzing data. This is a step that deserves due diligence in the planning process. The best standard plays a role in guiding the evaluator’s decisions in how to analyze and interpret data to assure that all stakeholder values are respected in the process of drawing conclusions. A timeline that transparently demonstrates inclusion of stakeholders facilitates acceptance of evaluation results and use of information.

Use, distribution, and feedback plan: Plans for use of evaluation results, communications, and dissemination methods should be discussed from the beginning. This is a critical but often neglected section of the evaluation plan. A communication plan that displays target audience, goals, tools, and a timeline is helpful for this section.

Checklist for ensuring effective evaluation report

• Provide interim and final reports to intended users in time for use

• Tailor the report content, format, and style for the audiences by involving audience members

• Include an executive summary

• Summarize the description of the stakeholders and how they were engaged

• Describe essential features of the system (e.g., in appendices)

• Explain the focus of the evaluation and its limitations

• Include an adequate summary of the evaluation plan and procedures

• Provide all necessary technical information (e.g. in appendices)

• Specify the standards and criteria for evaluative judgments

• Explain the evaluative judgments and how they are supported by the evidence

• Include examples of research and evaluation tools used




• List both strengths and weaknesses of the evaluation

• Discuss recommendations for action with their advantages, disadvantages, and resource implications

• Ensure protections for program clients and other stakeholders

• Anticipate how people or organizations might be affected by the findings

• Verify that the report is accurate and unbiased

• Organize the report logically and include appropriate details

• Remove technical jargon

• Use examples, illustrations, graphics, charts and stories.

Activity: Go to this page online and read through the evaluation report guidelines.

http://www.uic.edu/depts/crwg/cwitguide/05_EvalGuide_STAGE3.pdf


48



Introduction

The evaluation results may not reach the intended audience with the intended impact just because they are published. An intentional communication and dissemination plan should be included in your evaluation plan. The planning stage is the time for the assessment system to address the best way to share the lessons learned from the evaluation. The communication-dissemination phase of the evaluation is a two-way process designed to support use of the evaluation results for system improvement and decision making. In order to achieve this outcome, an assessment system must translate evaluation results into practical applications and must systematically distribute the information or knowledge through a variety of audience-specific strategies.

Distribute the report

Based on the uses for your evaluation, you will need to determine who should learn about the findings and how they should learn the information. Typically, this is where the final report is published, and most assume the evaluation is done. However, if personal ownership of evaluation results is inserted here, such as through collaboration with an evaluation working group, the impact and value of the evaluation results will increase. The system and working group then take personal responsibility for getting the results to the right people and in a usable, targeted format. This absolutely must be planned for and included in the evaluation plan. It will be important to consider the audience in terms of timing, style, tone, message source, method and format. Remember that stakeholders will not suddenly become interested in your product just because you produced a report. You must sufficiently prepare the market for the product and for use of the evaluation results. Writing a straightforward and comprehensive evaluation report can help ensure use.

Communicating evaluation results involves sharing information in ways that make it understandable and useful to stakeholders. Successful communication is the key to evaluation results being used. You can do this by using a variety of communication formats and channels. A communication format is the actual layout of the communication you will use, such as reports, brochures, one-page descriptions, newsletters, executive summaries, slides, and fact sheets. A communication channel is the route of communication you will use, such as oral presentations, videos, e-mails, webcasts, news releases, and phone conferences. Both the formats and channels should take into account the needs of different audiences, the type of information you wish to provide, and the purpose of the communication.

When developing your communication or dissemination strategy, carefully consider the following:

• With which target audiences or groups of stakeholders will you share findings?

• What formats and channels will you use to share findings?

• When and how often do you plan to share findings?

• Who is responsible for carrying out dissemination strategies?




The first step in writing an effective communications plan is to define your communication goals and objectives. Given that the communication objectives will be tailored to each target audience, you need to consider with the working group who the primary audience(s) are (e.g. the funding agency, the assessors, or some other group). Some questions to ask about the potential audience(s) are the following:

• Who is a priority?

• What do they already know about the topic?

• What is critical for them to know?

• Where do they prefer to receive their information?

• What is their preferred format?

• What language level is appropriate?

• Within what time frame are evaluation updates and reports necessary?

Once the goals, objectives, and target audiences of the communication plan are established, you should consider the best way to reach the intended audience by considering which communication or dissemination tools will best serve your goals and objectives. Will the program use newsletters/fact sheets, oral presentations, visual displays, videos, storytelling, and/or press releases? Carefully consider the best tools to use by getting feedback from your evaluation working group, by learning from others’ experiences, and by reaching out to target audiences to gather their preferences. An excellent resource to facilitate creative techniques for reporting evaluation results is Torres, Preskill, and Pionteck’s (2004) Evaluation Strategies for Communicating and Reporting.

Complete the communication planning step by establishing a timetable for sharing evaluation findings and lessons learned. The following matrix may help you plan your communication process.

What do you wnat to communicate?

To whom to you want to communicate?

How do you want to communicate?

Format(s) Channel(s)

This tool can help you track communications with your various audiences, including the communication format(s) (the layout of the communication, such as newsletters) and the communication channel(s) (the route of communication, such as oral presentations), audience feedback on the communication message, and next steps you need to take in response.

Communication Date Communication format(s)

Communication channel(s)

Stakeholder feedback and next steps


50


Ensuring Use

Communicating results is not enough to ensure use of evaluation results and lessons learned. The evaluation team and program staff need to proactively take action to encourage use and wide dissemination of the information gleaned through the evaluation project. It is helpful to strategize with stakeholders early in the evaluation process about how the assessment system will ensure that findings are used to support program improvement efforts and inform decision making. System staff and the evaluation working group must take personal responsibility for guaranteeing the dissemination of and application of evaluation results.

There are several practical steps you can include in your evaluation plan to help ensure evaluation findings are used. These steps might contain plans to:

• Conduct regularly scheduled meetings with evaluation stakeholders as a forum for sharing evaluation findings in real time and developing recommendations for program improvement based on evaluation findings

• Review evaluation findings and recommendations in regularly scheduled staff meetings

• Engage stakeholders in identifying ways they can apply evaluation findings to improve their programs

• Coordinate, document, and monitor efforts program staff and partners are making to implement improvement recommendations

• Develop multiple, tailored evaluation reports to address specific stakeholders information needs.





Introduction

Before you plan a presentation, it is critical to understand the people who you are presenting to. It is important to define your audience so that you can deliver a presentation that is meaningful and interesting. Not understanding your audience can lead to frustration and boredom for the listeners. There are some questions that you should always answer before planning the content of your presentation. Selecting the best information for your presentation is critical for the overall success of the talk. The content information must be relevant to the presentation itself and for the audience. Choosing the wrong information could lead to the audience becoming bored and disinterested. When selecting relevant information for your presentation it is important to be clear about the outcome you want from your presentation. When choosing information the first question you should ask is “What am I trying to achieve with my presentation?” This will help you choose useful and relevant information for your talk.

Questions for defining audience

It is important to know the background of your audience, the purpose for them coming to your presentation and the context in which you are delivering the presentation. The following checklist can be used to determine who your audience is.

Questions

• What is the group size? How many people will be attending the presentation?

� The number of people will affect many aspects of your presentation including your delivery style, how you present visual information, the amount and type of handouts or materials needed, and how much audience participation or interaction is possible

• What is the average age of the audience? Is there a wide variety of ages represented or are all participants of a similar age?

� If the audience is predominantly made up of one age range such as younger or older, this will affect the types of examples you may use for your presentation

• What jobs or positions do the people in the audience hold? How does this relate to your position?

� If the audience is composed of your superiors, the language and tone you use will be important. You would have to use more formal and professional language

� If your audience is predominantly made up of your peers, you may use more informal language that they can relate to. It is also important not to use language that could be seen as condescending for people with lower ranks than you

• Why are people joining your presentation?

� If they have been instructed to attend your presentation, the audience may not be as engaged in comparison to an audience that has been invited or who attend because of their own interest


52


• How much do the audience know about your topic?

� If your audience is familiar with the information, you will not have to provide much background or explanation

� If your audience is unfamiliar with the topic, you will have to provide some support for them

� You can always ask questions at the beginning of or before your presentation, to find out how much background knowledge your audience has about the topic

• Where are you presenting? What time is the presentation?

� Is the room air conditioned/heated? Is it small, large, hot, cold or loud?

� Has the audience been working all day or is it being presented very early in the morning?

� These things will impact on your audience’s enthusiasm and engagement. If they are tired, hot and uncomfortable, you might have to adjust your presentation to recognize these conditions.

Understanding your audience

There are many things that you can do to align your presentation with audience expectations. Using appropriate language for your audience is critical to make it successful as well as presenting a topic that is interesting or engaging for your audience. Understanding your audience will help you to decide what type of information and language you will use in the presentation.

What information do you need?

Finding relevant and useful information for your presentation is often the most challenging part of the process. It is common to either have too much information or not enough and this can lead to difficulties in preparing your presentation. It is important to think about your audience when selecting information for your presentation. If your audience does not know anything about your topic, you will have to include more information to explain the topic. If your audience understands the topic clearly, you will need to choose information that supports the goal of your talk which may be to argue a point of view or new idea.

When selecting relevant information you should:

• Know who your audience is and what they expect from your presentation

• Have a clear purpose or goal for your presentation

• Know how long your talk will go for

• Write an outline for your presentation

• Understand clearly the context or why you are presenting

• Choose information from appropriate sources that relates to your outline.




Selecting relevant information

When you have completed your outline you need to match relevant information with your main points. Different types of presentations will require different types of information. Look at the examples below. Can you think of more examples of relevant information for each type of presentation?

Reports Informative Instructional Arousing

Facts

Figures

Graphs, charts, diagrams

Survey results

Data

Recommendations

Questions

Facts

Time: when things should be done

Place: where things should happen

Cause and effect: how things should happen

Diagrams and visual aids

Provide specific instructions or orders

Why is the information valuable?

Process and steps

Demonstrate

Visual aids

Explanation of new terms or equipment

Questions and feedback

Raise attention

Examples or stories

Comparison of past and present

Audience experiences

Feedback and questions

Persuasive Other Other Other

Facts and examples

Visual aids and data

Potential solutions or benefits

Reasons why

Questions

A successful presentation

An introduction Speaker introduces them self and the topic

An outline Speaker briefly explains what he is going to talk about

Main body Speaker gives the main points of his presentation

Summary Speaker repeats the main points of his presentation in a shortened form

Conclusion Speaker ends his presentation and thanks the audience for listening

Questions Speaker asks the audience if they have any questions about his presentation


54


How to respond to questions effectively

Many presenters, especially non-native English speakers, are scared of the question and answer section after they have made their presentation. The question session is an important part of the whole presentation and it provides an opportunity for the presenter to clarify any parts of the talk that the audience did not understand. Planning for questions is an effective way to prepare for this section of the presentation and can determine the overall success or failure of the talk. There are many steps that you can take to prepare for questions.

To prepare for questions about your presentation you should:

• Research your topic thoroughly

• Determine the types of question you may be asked and prepare responses for these questions

• Make sure you have organised enough time for questions within your presentation timing

• Ask the audience if they have any questions

• Draw the audience’s attention to the focus of your topic and frame the question parameters, for example “do you have any questions about the three issues I raised?”

Steps for responding to questions

1. Listen – When responding to questions it is important to listen fully to the question first before attempting a response. Often a question will change whilst the questioner is speaking or the question may come right at the end of the interaction. Do not preempt the question.

2. Clarify – Sometimes it is difficult to undertsand exactly what the question is or there may seem to be more than one question being asked. You may, therefore, have to ask for clarification. You can rephrase the question “So you would like me to explain...?” or you can check that you understand what the question is “Do you mean...?” or “Would you like me to explain ...... or ...........?”.

3. Include all the audience – It is important to keep all of the audience involved in the question section of your presentation as others may have had a similar question. Make sure everyone has heard the question and speak to the whole audience when you respond not only the questioner.

4. Respond – Try to respond to the question in the most succinct and easiest way. Do not go off topic and introduce new information that may confuse the audience. Restate information form your presentation in another way and check with the audience to see if your answer is satisfactory “Is that OK?” or “Does that answer your question?”

5. Include a mechanism for follow-up questions – If your time is limited and there are still more questions, allow people to email you with any other questions they may have. Make sure you respond to them!





Introduction

Justifying conclusions includes analyzing the information you collect, interpreting what the data means, and drawing conclusions based on the data. This step is needed to turn the data collected into meaningful, useful, and accessible information. This is often the step in which programs incorrectly assume they no longer need the stakeholder workgroup and that this step is better left to the “experts.” However, including your stakeholder group in this step is directly tied to the previous discussion on credibility and acceptance of data and conclusions.

Moreover, it is critical that your plans allow time for interpretation and review from stakeholders (including your critics) to increase transparency and validity of your process and conclusions. The emphasis here is on justifying conclusions, not just analyzing data. This is a step that deserves due diligence in the planning process. The propriety standard plays a role in guiding the evaluator’s decisions on how to analyze and interpret data to assure that all stakeholder values are respected in the process of drawing conclusions. This may include one or more stakeholder interpretation meetings to review interim data and further refine conclusions.

Gathering feedback about the report

A variety of activities can be included in your evaluation plan to solicit stakeholder feedback and facilitate interpretation of evaluation data. An example is provided below:

Interpretation and Feedback Activities Timeline

Individual site evaluation reports and feedback Within 1 month of site visit

Check-in with Evaluation Working Group and/or participants

Within month of site visit or as appropriate during analysis phase

Evaluation interpretation meeting

Immediately following preparation of preliminary results

Stakeholder interpretation meeting

Within 1 month following the evaluation interpretation meeting

Stakeholder review of draft final report

Within 1 month following the stakeholder interpretation meeting

Clearance and review process of final report

Within 1 month following stakeholder review of draft final report


56


It is important to consider the time it takes to solicit and incorporate stakeholder feedback in your evaluation project timeline. At this time, you should revisit your budget and timeline created earlier to ensure adequate time and funding for the stakeholder feedback process.

In order to make sure your stakeholder interpretation meeting is a success, plan for steps to help things run smoothly. Time for these activities needs to be included in your evaluation timeline.

• Send the initial invitation at least 2 months in advance so that stakeholders can plan for the meeting. Remind stakeholders of the overall evaluation purpose and questions

• Send the preliminary report or PowerPoint presentation within 2 weeks of the initial invitation to allow stakeholders time to review. It is important to remind stakeholders that results are a draft and should not be shared outside of the review group

• Send reminders about the meeting 1 or 2 weeks prior to the date. Identify any pre-existing documentation that may be useful for understanding context

• Plan for appropriate technology (and backup) needed such as recorders, laptop, and screen, flipcharts

• If feasible, use a professional meeting facilitator.

In order to gather feedback from stakeholders, there are various ways that feedback could be obtained. These include:

• Verbal feedback at various stages through the evaluation or at the stakeholder meeting and presentation of report

• Written, printed, or electronic feedback in response to information shared throughout the evaluation or in relation to the draft report

• Structured feedback which could be organised with feedback forms or questionnaires, focus groups, an meetings

• Feedback from managers or key stakeholders through approvals to proceed with different stages of the evaluation process.





Introduction

After engaging in feedback activities with stakeholders, it is necessary to respond to the feedback provided in order to improve the system as a whole. The recommendations that have been presented might be satisfactory or there may be further suggestions from stakeholders that are valid and useful. It is important to analyse the feedback given and if there are sufficient respondents regarding similar feedback, as well as the feedback aligning with evaluation goals, then the draft recommendations need to be adjusted accordingly to include feedback suggestions.

Revise draft recommendations

As with gathering evidence for the evaluation, altering draft recommendations based upon feedback should go through a similar process of analysis. Is the feedback sufficient? Is it relevant to the evaluation purposes or assessment system goals? Is the feedback appropriate and able to be implemented?

Just changing recommendations based upon feedback is not effective. The feedback needs to be well thought out and useful to the overall evaluation. Not all stakeholders will have the same amount of access to information and data as the evaluation team, so some of their feedback for change may be inadequate. On the other hand, stakeholders with limited access to information may have insights that have not been thought of by people who have been heavily involved in the evaluation process.

Any recommendations need to be:

• Clearly numbered and linked to the evaluation question

• Based upon evidence that is sufficient and valid

• Made available to relevant stakeholders in order for them to provide feedback

• Reviewed by stakeholders and any changes agreed upon

• Any changes to draft recommendations need to be documented and approved by relevant stakeholders.


58



Introduction

Now that the evaluation has been completed, feedback sought, and revision of recommendations carried out, the evaluation process and data gathered needs to determine what action needs to be taken to improve or cease with the assessment system. The actions need to be related to the evaluation recommendations, which are in turn aligned with evaluation findings and goals. In determining what action needs to taken, it is important to make sure that the intended action meets the needs of the recommendations and assessment system, and is feasible. For example, there is no point in determining that an external assessment team needs to be hired, if there is no budget for this action.

What action is needed?

There are many different actions that may be deemed necessary on completion of an evaluation of the assessment system. These actions will be specific to the assessment system that has been evaluated and they could include:

• Continuing with the existing assessment system arrangements

• Modifying assessment arrangements with existing internal and/or external assessment providers

• Discontinuing the existing assessment system arrangements

• Moving internal assessment to an external provider and/or moving external assessments to an internal system

• Changing external assessment providers

• Communicating satisfaction and/or dissatisfaction with assessment service providers

• Placing additional specific service requirements on assessment service providers

• Altering the internal assessors used, or providing existing assessors with identified training to address deficiencies in their practice

• Modifying existing assessments system components due to findings from the evaluation including changing assessment methods, venues, timings, assessment tools.

Activity: In groups look at some evaluation questions and areas of evaluation of the assessment system and predict some potential recommendations for action that may result from an evaluation. What action would need to be taken to address the recommendations?




Action plan

Once the actions have been determined, an action plan will need to be developed. The action plan could follow these steps:

Define the problem (actions that need to be taken)

1. Brainstorm ideas

2. Prioritize ideas

3. Develop problem statements(s)

Develop solutions

1. Brainstorm ideas

2. Prioritize ideas

3. Develop problem statements(s)

Develop a tentative action plan

1. Brainstorm ideas

2. Prioritize ideas

3. Examine possible obstacles to plan implementation

4. Develop ways to remove any obstacles or develop alternative plans

Develop final action plan which includes

1. All components (parts of the plan)

2. Individual assignments

3. Time frame

4. Budget needs

5. Evaluation method

6. Date for final evaluation

Action plan template:

Action Needed

Estimated Time

Frame

Actual Time

Frame

Budget $ Spent

Action taken

Evaluation Further Action Needed


60


Work Projects


1.1 Remember to include all of the necessary parts including:

• Title page, introduction, evaluation purpose statement, definition of the assessment system, which part you are evaluating etc. Go through Performance Criteria 3.1 to help you.

3.2. Complete a communication matrix that identifies who needs access to the report, why they need it, and how you will supply access to the report. Show the matrix to your trainer.

3.3. Prepare and deliver a 5 minute oral report to your class on the findings and results of your evaluation.

3.4. Ask your classmates for feedback on your report and colleagues from work. Identify how you will gather their feedback and explain the feedback gathering activities to your trainer.

3.5 Develop a set of recommendations based upon the results of your evaluation and distribute them to relevant stakeholders in your organisation. After gathering feedback, identify the changes that may need to be made to your recommendations and write the revised recommendations, making sure they align to the goals of the evaluation. Show the revised recommendations to your trainer and explain what changes were made due to feedback received.

3.6 Write an action plan that identifies the action to be taken as a result of the evaluation. Use an action plan matrix and show it to your trainer.




Summary

Prepare an evaluation report

The evaluation report is the summary of all the work that has been done to gather data and respond to it. It is important that the evaluation report is clear, accessible, and allows opportunities for feedback. Including stakeholder contribution in all parts of the evaluation is an important aspect of an effective evaluation, and providing opportunities for them to respond to the report is critical in getting buy-in to the results and recommendations. The evaluation report needs to be accessible to all relevant stakeholders and this may be achieved through:

• Stakeholder meetings

• Electronic media: emails, websites, online summaries

• Oral presentation of the report.


62


Presentation of written work




1. Introduction

It is important for students to present carefully prepared written work. Written presentation in industry must be professional in appearance and accurate in content. If students develop good writing skills whilst studying, they are able to easily transfer those skills to the workplace.

2. Style

Students should write in a style that is simple and concise. Short sentences and paragraphs are easier to read and understand. It helps to write a plan and at least one draft of the written work so that the final product will be well organized. The points presented will then follow a logical sequence and be relevant. Students should frequently refer to the question asked, to keep ‘on track’. Teachers recognize and are critical of work that does not answer the question, or is ‘padded’ with irrelevant material. In summary, remember to:

• Plan ahead

• Be clear and concise

• Answer the question

• Proofread the final draft.

3. Presenting Written Work

Types of written work

Students may be asked to write:

• Short and long reports

• Essays

• Records of interviews

• Questionnaires

• Business letters

• Resumes.

Format

All written work should be presented on A4 paper, single-sided with a left-hand margin. If work is word-processed, one-and-a-half or double spacing should be used. Handwritten work must be legible and should also be well spaced to allow for ease of reading. New paragraphs should not be indented but should be separated by a space. Pages must be numbered. If headings are also to be numbered, students should use a logical and sequential system of numbering.


64


Cover Sheet

All written work should be submitted with a cover sheet stapled to the front that contains:

• The student’s name and student number

• The name of the class/unit

• The due date of the work

• The title of the work

• The teacher’s name

• A signed declaration that the work does not involve plagiarism.

Keeping a Copy

Students must keep a copy of the written work in case it is lost. This rarely happens but it can be disastrous if a copy has not been kept.

Inclusive language

This means language that includes every section of the population. For instance, if a student were to write ‘A nurse is responsible for the patients in her care at all times’ it would be implying that all nurses are female and would be excluding male nurses.

Examples of appropriate language are shown on the right:

Mankind Humankind

Barman/maid Bar attendant

Host/hostess Host

Waiter/waitress Waiter or waiting staff

Recommended reading



Recommended reading

Making your ideas stick, reporting and program planning

Atkinson C. Beyond Bullet Points: Using Microsoft Office PowerPoint 2007 to Create Presentations That Inform, Motivate, and Inspire. Microsoft Press, 2007

Becker HS. Writing for Social Scientist: How to Start and Finish Your Thesis, Book, or Article. Chicago, IL: University of Chicago Press, 2077, 2nd ed

Heath C, Heath, D. Made to Stick: Why some Ideas Survive and Others Die, New York, NY: Random House, 2007

Heath C, Heath D. Switch: How to Change Things When Change is Hard. New York, NY: Random House, 2010

Lavinghouze R, Price AW, Smith, KA. The Program Success Story: A Valuable Tool for Program Evaluation. Health Promotion Practice, 2007; 8(4): 323–331

Torres R, Preskill H, Piontek ME. Evaluation Strategies for Communicating and Reporting. Thousand Oaks, CA: Sage Publications, Inc., 2004, 2nd ed

Qualitative methods

Patton M.Q. Qualitative Research and Evaluation Methods. Thousand Oaks, CA: Sage Publications, Inc., 2001, 3rd ed

Yin RK. Case Study Research: Design and Methods (Applied Social Research Methods) Thousand Oaks, CA: Sage Publications, Inc., 2008, 4th ed

Yin RK. Qualitative Research from Start to Finish. New York, NY: The Guilford Press, 2010

Quantitative methods

Kleinbaum DG, Klein M. Logistic Regression: A Self-Learning Text (Statistics for Biology and Health. Springer, New York, NY, 2010, 3rd ed

Rothman KJ, Greenland S, Lash TL. Modern Epidemiology. Philadelphia, PA: Lippincott Williams & Wilkins, 2008, 3rd ed

Tufte ER. The Visual Display of quantitative Information. Cheshire, CT: Graphics Press, 2001

Evaluation use

Butterfoss FD. Coalitions and Partnerships in Community Health. San Francisco, CA: Jossey-Bass, 2007

Mattessich PW. The Manager’s Guide to Program Evaluation: Planning, Contracting, and Managing for Useful Results. St. Paul, Minnesota: Amherst H. Wilder Foundation, 2003

Patton MQ. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York, NY: The Guilford Press, 2010

Patton MQ. Utilisation-Focused Evaluation. Thousand Oaks, CA: Sage Publication, 2008

Recommended reading

66


Useful websites:

Assessment systems: http://equella.think.edu.au/lor/file/c5c75dd7-cfb6-63ba-de02-74a178fbb10b/7/AssessmentPolicyAndProcedure.pdf

Evaluation ideas: http://www.eval.org

Impact and Value: Telling Your Program’s Story: www.cdc.gov/oralhealth/publications/library/success_stories_wkbk.htm

Trainee evaluation sheet



Trainee evaluation sheet


The following statements are about the competency you have just completed.

Please tick the appropriate box Agree Don’t Know

Do Not Agree

Does Not Apply

There was too much in this competency to cover without rushing.

Most of the competency seemed relevant to me.

The competency was at the right level for me.

I got enough help from my trainer.

The amount of activities was sufficient.

The competency allowed me to use my own initiative.

My training was well-organized.

My trainer had time to answer my questions.

I understood how I was going to be assessed.

I was given enough time to practice.

My trainer feedback was useful.

Enough equipment was available and it worked well.

The activities were too hard for me.

Trainee self-assessment checklist

68


The best things about this unit were:

___________________________________________________________________

___________________________________________________________________

___________________________________________________________________

___________________________________________________________________

The worst things about this unit were:

___________________________________________________________________

___________________________________________________________________

___________________________________________________________________

___________________________________________________________________

The things you should change in this unit are:

___________________________________________________________________

___________________________________________________________________

___________________________________________________________________

___________________________________________________________________





As an indicator to your Trainer/Assessor of your readiness for assessment in this unit please complete the following and hand to your Trainer/Assessor.


Yes No*















2.4 Store data



70


Yes No*








Statement by Trainee:

I believe I am ready to be assessed on the following as indicated above:

Signed: _____________________________

Date: ____________

Note:

For all boxes where a No* is ticked, please provide details of the extra steps or work you need to do to become ready for assessment.


72


tm evaluate effectiveness of assessment system refined€¦ · the association of southeast asian...

Documents