thinking evaluatively evaluation framework for measuring program outcomes

Thinking Evaluatively

Evaluation Framework for Measuring Program Outcomes

Nav GhimireAgriculture Agent, Green Lake County

Team Member, ELST

Judgment about how well your

program objectives are being met….

What is Evaluation ?

Nav Ghimire, Green Lake County Agriculture Agent, 2014

Evaluation is…….

Developing Criteria

Gathering Evidence

Forming Judgments


Criteria The measure against which judgments are made about.

“Outcome/Program Objective / Evaluative Question”

EvidenceData that is relevant to evaluative question to make decision.

May be obtained from a variety of measurement tools, survey, interview, observations, reports, records, etc.


JudgmentJudgments should be made based upon the degree to which

the evidence indicates that the criteria are being met.

Judgment About What ? * Appropriateness – Looks at process than products. - Were participant needs met?

- Was what learned important to them? - Were teaching methods appropriate?

* Effectiveness – Measures program accomplishments in terms of changes in participant behavior. - Knowledge, skills , attitude, and application of practice?

* Efficiency – Compares time, resources, and effort expended in terms of accomplishments. - Was accomplishment sufficient for resources used?


Process Evaluation Outcome Evaluation Impact Evaluation

How is program/ training implemented?

Ability to rigorously collect data on the program outcomes

Collect data on program impacts

Are activities delivered as intended? Changes in participant’s

Knowledge

Changes in

Economic

Are participants being reached as intended?

Skills Social

What are participant reactions?

Attitude Environmental conditions of the individuals and community

Was teaching method appropriate?Aspiration

Educator’s self–assessment?

Behavior – application /change of practice

Steps in Evaluative Thinking

Source: Journal of Agricultural Education Volume 53, Number 4, pp 85–97; UW-Extension, PD& E


Process Evaluation How is program/ training implemented? e.g., venue, training hall, noise, seating arrangements, food, timing etc.

Are activities delivered as intended?

e.g., curriculum, content etc.

Are participants being reached as intended?

e.g., targeted audience, number of participants etc.

What are participant reactions? e.g., educator’s knowledge, skills, presentation style, learning material, delivery style.

Educator’s self–assessment?

e.g. what went wrong, how can I improve next time etc.


Outcome Evaluation Usually ConductedAbility to rigorously collect data on the

program outcomes

Changes in participant’s

KnowledgeImmediately after the program

Skills Immediately or after some time

Attitude After some time

Aspiration After some time

Behavior – application /change of practice

After some time


Impact evaluation Usually Conducted

Collect data on program impacts

Changes in - Economic After few years

Social After few years

Environmental conditions of the individuals and communities

After few years


List intended outcomes from the program.

Write measurable program objective(s) to reach each outcome.

List program activities to achieve each objective.

Write evaluative question(s) aligned to each objective.

Identify key variable(s) that evaluation question(s) contain.

Operationalize the variable(s) and identify indicator(s) to assess

Identify source(s) of data.

Determine data collection method(s).

How to Measure Program Outcomes


How to Write a Good Objective

Specifies a result, not an activity.

Each objective describes just one result.

Emphasize what will be done but does not explain how to do it.

It is specific, measurable, achievable, possible, verifiable.

Should have a subject, verb, and object (SVO)

It is designed with and understood by those responsible for its

implementation - Extension educators based on needs assessment.


Outcome: Number of farmers developing a nutrient management plan for their farm will be increased.

Objective 1: Farmers will gain knowledge and skills about phosphorous management. Correct: Farmers will gain knowledge about the importance of phosphorous management in their farm /soil.

Objective 2: Farmers will learn skills to manage phosphorous in their farm

Objective 3: Farmers will learn skills to manage potassium in their soil.

Objective 4: Farmers will gain knowledge about the importance of a nutrient management plan for their farm. Objective 5: Farmers will learn skills for writing a nutrient management plan for their farm.

Example of Nutrient Management Program


Outcome: Numbers of parents/caregivers developing nurturing abilities to effectively raise their children will be increased

Objective 1: Parents/ caregivers will gain knowledge about the importance of nurturing to support their children.

Objective 2: Parents/caregivers will learn nurturing skills to support their children / effectively raise their children.

Outcome: Number of youth developing abilities to resist negative peer pressure will be increased.

Objective 1: Youth will gain knowledge about the consequences of negative peer pressure in the school.

Objective 2: Youth will learn skills to resist negative peer pressure to avoid bad habits in the school / life.

Impact: Number of better citizen in the community will be increased.

Example of Family Living / Youth Program


Program Objective: They are learning objectives that you want your participants to learn as a result of your program. It guides program activities you want to implement as an educator to facilitate learning.

Program Outcomes: What participants will achieve / are going to do as a result of your program / educational activities.

Evaluation:Measures to what extents program achieved its objectives to produce a desired outcomes.

Program objectives Vs Programs Outcomes


Evaluative QuestionsWhat is it that you want to know about your program?

operationalize it (make it measurable)

Do not move forward if you cannot answer this question.


Evaluative Questions

Provides guidelines to direct your evaluation

Demonstrates what area of programming outcomes you are interested to know about Primarily not designed as a question to put in survey

Can be a part of the survey question, if needed


Question 1: Did farmers increase knowledge about the importance of phosphorous management in their soil?.

Correct Q.1: How much did program / training increased farmers’ knowledge about the importance of phosphorus management?

Question 2: Did farmers develop skills for writing a nutrient management plan for their farm?

Correct Q.2: To what extent the training / program developed farmers’ capacity to write a nutrient management plan for their farm?

Question 3: How well did farmers correctly developed nutrient management plans for their farm?

Question 4: How well did farmers applied nutrient management plan in their farm.

Question 5: How much did farmers saved cost from improved nutrient management practices?

Evaluative QuestionsNutrient Management Training Program


Question 1: How much did program/training increased parents / caregivers knowledge about the importance of nurturing?

Question 2: How well did parents / caregivers applied nurturing skills to raise their children?

Question 3: To what extent youth developed their capacities / abilities to resist negative peer pressure ?

Question 3: How well did youth applied negative peer pressure skills to avoid bad habits?

Question 4: How much could youth able to resist negative peer pressure to avoid bad habits ?

Evaluative Question Family Living / Youth Program


How well the program / training was implemented?

To what extent has the program / training contributed to meet the needs of participants.

How much did program / training increased participants’ knowledge and skills?

To what extent the program / training was enough to solve participant’s farming problems?

Examples of Good Evaluative Question(s) ?


Evaluative Question Contd…

How significant was the program’s contribution to strengthening the partner organization(s)?

To what extent did the program work well to all participants?

To what extent the possible program’s positive results are likely to last after the project ends?


Variable Indicators/ Measure Data Source

Q1. Knowledge

gain on P management

Number of farmers gained knowledge in phosphorous management

Surveying participants

Q2. Nutrient Management plan developed

Number of farmers developing nutrient management plan


Checking with Land and Water conservation to see how many farmers submitted the plan

Evaluation FrameworkNutrient Management Training Program


Variable Indicators/ Measure Data Source

Q1. Knowledge gain on nurturing

Number of parent/caregivers gained knowledge in nurturing skills


Q2. Ability to resist negative peer pressure

Number of youth increased ability to resist peer pressure 1. Surveying participants

2. Surveying parents / school teachers /mentors.

Evaluation FrameworkFamily Living /Youth Program


Use self-constructed questionnaire

Each question should focus on one idea only

Easy to answer (Check rather than write), if possible

Keep the same scale throughout the questionnaire

Maintain floor and celling in the scale; very low – very high

Ask only what you need to meet your objective(s)

Use open ended questions when you want details

Give your friends / farmers to read and advise you

Questionnaire for Data Collection


Circle the appropriate number to indicate change in your knowledge on the listed topics after you participated in Nutrient Management Training

1 = Very low 2 = Low 3 = Moderate 4 = High 5 = Very high Topic Areas

Before Training

After Training

How to sample soil 1 2 3 4 5 1 2 3 5 5

Apply nutrient recommendation from soil test report

1 2 3 4 5 1 2 3 5 5

How pH affects nutrient availability

1 2 3 4 5 1 2 3 5 5

Crediting N from manure and legumes

1 2 3 4 5 1 2 3 5 5

P availability from fertilizers and manure

1 2 3 4 5 1 2 3 5 5

An Example of Questionnaire


Types of Question for A Program Evaluation Survey

Reactions to the Training / ProgramPlease rate the training on the following items:

Items Poor Fair Average Good Very Good

Content 1 2 3 4 5

Organization 1 2 3 4 5

Teaching 1 2 3 4 5

Involvement of participants

1 2 3 4 5

Workbook Material


Types of Question Contd…..

Perception / Attitude

Please indicate to what extent do you agree with the todays’ training

Items Agree Strongly agree

Disagree Strongly Disagree

N/A

Stimulated me to think 1 2 3 4 5

Motivated me to learn more 1 2 3 4 5

Motivated me to do something different

1 2 3 4 5

Motivated me to talk about this with my friend

1 2 3 4 5



Confidence Please indicate your level of confidence to perform the skills that you learned form the training:

Items Not confident

A little confident

Somewhat confident Confident

Highly confident

Managing phosphorous in soil 1 2 3 4 5

Writing a nutrient management plan for my farm

1 2 3 4 5

Applying the nutrient management plan in my farm

1 2 3 4 5

Helping my friend to develop his nutrient management plan

1 2 3 4 5



Intentions As a result of this training, please indicate your intentions for applying learned skills / information in your farm / family / work:

Items No Plans to apply

May apply

Will definitely apply this

Need More

Help To apply this

Already applying this



1 2 3 4 5


1 2 3 4 5

Helping my friend to develop his nutrient management plan

1 2 3 4 5



Behavior ChangeTo what extent did you change the following practice that you learned from the training /program:

Items None 10% of my farm

25% of my farm

50% of my farm

100% of my farm



1 2 3 4 5


1 2 3 4 5


Workshop

1. Identify a program.

2. Write 1 intended outcome.

3. Write 2 program objectives.

3. Write 1 evaluative question.

4. Display your work on the wall.


thinking evaluatively evaluation framework for measuring program outcomes

Documents

program training

time nav ghimire

judgments nav ghimire

program impactschanges

program skillsimmediately

program outcomeschanges

program objectives

green lake county team