the internal learning system, assessing impact while empowering internal learning

THE INTERNAL LEARNING SYSTEM ASSESSING IMPACT WHILE EMPOWERING INTERNAL LEARNING

Anna Kiel Martin Katrina Mitchell

American Evaluators Association National Conference, Minneapolis October 26, 2012

Sel

f Hel

p G

roup

Pro

mot

ion,

PR

AD

AN

. Pho

to s

ourc

e: H

elzi

Nop

onen

•  SIMPLE

•  Quantitative

•  Evaluation +

•  Non-extractive

•  Highly participatory

•  Pictorial/accessible

•  Customized

•  Embedded in the program

•  Multi-year

WHAT IS ILS?

HOW DID ILS DEVELOP?

Helzi Noponen in Bihar, 2006. Photo credit: Valerie Arendt

WHY USE ILS?

EXERCISE: HOW DOES THIS WORK?

You have been given a sample page.

Identify what the purpose of the page is and how you would interact with it as a participant.

How would you use it to assess impact or empower learning?

Social ProblemsSanitation

EVALUATION

LEARNING

?

“Suicide is Not a Solution”

PLANNING

Livelihood Production Plan

HPWDS

?

Problem Sorting

PLANNING Strategic Interests Plan

?

WHAT’S IN THE ILS TOOLBOX?

“Women own the ILS book. It is their record. Though illiterate, they can “read” it. No one has ever before suggested they can do such work. They take the responsibility seriously.

It is like a green light for them to dare to think about achieving a better life.

Each indicator picture has an implied program value attached to it.

Women absorb this.

It gives permission or space to discuss sensitive empowerment and wider social issues and the courage to attempt change”

Helzi Noponen

HOW DO YOU KNOW YOUR CURRENT SITUATION?

Evaluation formats:

•  Yes/no questions

•  Quantity questions

•  Multiple choice questions

•  Ranking and/or scaling questions

General Health Care Practices

Woman goes to doctor when needed. Safe water practices

Knows oral re-hydration therapy. Washes hands after latrine and before food handling.

Hygiene around menses

Woman herself uses mosquito net.

General Health Care Practices

Woman goes to doctor when needed. Safe water practices

Knows oral re-hydration therapy. Washes hands after latrine and before food handling.

Hygiene around menses

Woman herself uses mosquito net.

HOW DO YOU IMAGINE A BETTER FUTURE?

Learning formats:

•  Bad Scene / Good Scenes

•  Cautionary Cartoons / Illustrated Folk Tales

•  Dream Scenes

•  Panorama Scenes

•  6 panel Stories (photo novella, picture parade, comic strip)

?

“Suicide is Not a Solution”

HOW WILL YOU CREATE A BETTER FUTURE?

Planning formats:

•  Problem Sorting Exercises

•  Goal Setting Exercises

•  Priority Choosing Exercises

•  Planning Formats – budgeting trees, loan repayment plans

Plan for Meeting Strategic Interests

think discuss husband family help friend help get comfort

other action get service

meet officials work with men

write officials research

elite help campaign

cluster help SHG help

other SHG help protest action exposure visit speak out training

0 1 2 3

HOW DO YOU WEAVE IT TOGETHER?

Learning

Evaluation

Planning

EXERCISE: HOW DO YOU… You are going to a pot-luck and have to bring a dish.

What mental steps do you go through from the invitation to arriving at the right place, at the right time, with the right food to share?

PARTICIPANT: HOW DOES ILS CHANGE BEHAVIOR?

2. Assess

3. Analyze

4. Plan or alter

5. Document, share, reinforce

1. Collect

5 Tasks

PARTICIPANT: HOW DO I USE THIS TO IMPROVE MY SITUATION?

Dalit women in Varanasi displaying their job cards Photo source: Leena Patel/UN Women (seeking permission)

•  5 tasks of ILS

•  Ownership of diary/data – participatory

•  Census – they all have one

•  Community-building, decreases isolation, empowers (story telling)

HOW DOES A PARTICIPANT INTERACT WITH ILS?

WHO INTERACTS WITH ILS?

PARTICIPANT PROGRAM STAFF

EVALUATOR AGENCY/ORGANIZATION Program participants with diaries/workbooks PRADAN, Photo source: Helzi Noponen Helzi Noponen in Bihar, 2006. Photo credit: Valerie Arendt

Field promoter with PRADAN, Photo source: PRADAN Image source: PRADAN website

•  Facilitation in a group setting

•  Reveals conditions/behaviors which aren’t readily visible

•  Highlights patterns

•  Demands radical accountability

•  Encourages reflective practice and innovation

HOW DOES A PROGRAM MANAGER INTERACT WITH ILS?

•  How to (steps to ILS):

• Design

•  Implementation

• Measurement

•  Commitment to participatory techniques

•  Effective creation and implementation mirrors the tool; lots of learning and adapting

HOW DOES AN EVALUATOR INTERACT WITH ILS?

•  Impact assessment

•  Can quantify hard-to-measure social change outcomes

•  Use with outside stakeholders for funding and advocacy

•  Commitment to participatory techniques, accountability and innovation/learning

HOW DOES THE AGENCY INTERACT WITH ILS?

HOW WOULD YOU USE IT?

EARLY CHILDHOOD HEALTH CARE

BUSINESS DEVELOPMENT HOMELESS / YOUTH Stock photo Recent immigrants. Photo courtesy of Neighborhood House

Maternal / neo natal health care. Photo courtesy of Neighborhood House Stock photo

IS ILS RIGHT FOR MY PROJECT / ORGANIZATION?

I want QUANTITATIVE data

Learning is important

Participant input is important

Formative and summative

Process and outcome data

Longitudinal, long-term

Highly customized

I need QUALITATIVE data

I just need to assess

Summative only

Outcome only

Short term or one-off evaluation

YES NO

USE “TRADITIONAL” M&E or EXTRACTIVE PIA

HOW WOULD YOU CONSTRUCT SOMETHING LIKE THIS…

….as it was originally meant to be used?

WHAT SHOULD I CONSIDER WHEN DESIGNING AN ILS DIARY?

Research design

•  Identify your needs and analysis plan before you start

Survey design

•  Both formative and summative

•  Types of questions

•  Level of detail

•  Relevant to all participants

Diary design

•  Pictorialize – lots of considerations!

•  Create modules

•  Start simple. Build up to complexity and risk.

HOW MUCH DETAIL? DEPENDS…

The semi-extractive interview event:

•  Side-by-side

•  Sharing

•  Transparency

•  Verification

•  Human Subjects Protocol

Land Utilization

optimum yield

above 75% yield

50% - 75% yield

25% - 50% yield

Less than 25% yield

20

number of yearly harvests?

base mid point end point


# grain harvests

grain yield



# pulses harvests

pulses yield



# oilseeds harvest

oilseeds yield



# vegetables harvest

vegetables yield



# fruits harvest

fruits yield

Include sugarcane1

2

3

4

5

average yield?

99=not farmer (skip page)

base mid end

base mid end

base mid end

base mid end

base mid end

Grows crop? 0=No; 1=Yes

Land Utilization

optimum yield

above 75% yield

50% - 75% yield

25% - 50% yield

Less than 25% yield

20




# grain harvests

grain yield



# pulses harvests

pulses yield



# oilseeds harvest

oilseeds yield




vegetables yield



# fruits harvest

fruits yield

Include sugarcane1

2

3

4

5

average yield?

Land Utilization

optimum yield

above 75% yield

50% - 75% yield

25% - 50% yield

Less than 25% yield

20


base mid point end pointbase mid point end point


# grain harvests

grain yield



# pulses harvests

pulses yield



# oilseeds harvest

oilseeds yield




vegetables yield



# fruits harvest

fruits yield

Include sugarcane1

2

3

4

5

average yield?

99=not farmer (skip page)

base mid end base mid end





Grows crop? 0=No; 1=Yes

Land Utilization

optimum yield

above 75% yield

50% - 75% yield

25% - 50% yield

Less than 25% yield

20

number of harvests

Land Utilization

optimum yield

above 75% yield

50% - 75% yield

25% - 50% yield

Less than 25% yield

20

number of harvests

HOW DO YOU TURN PICTURES AND PENCIL LINES INTO DATA?

PARTICIPANT DIARY INTERVIEWER DIARY

WHAT ARE ITS STRENGTHS?

Extractive Empowering METHOD

Donors/policy Clients AUDIENCE

Judging Learning PURPOSE

External Internal ORIENTATION

From Simonowitz, 2001

Traditional M&E Participatory Impact Assessment

COPYRIGHT, ATTRIBUTION AND USAGE

A note on copyright, attribution, and appropriate usage.

The Internal Learning System was developed by Dr. Helzi Noponen in partnership with Ford Foundation India and Imp-Act, and implemented with a number of organizations including PRADAN, the Handloom Weavers Development Society (HLWDS), ASA, the High-range

Plantation Workers Development Society (HPWDS), PLAN Honduras, and others.

Many of the illustrations contained in this presentation come from diaries/workbooks created by Dr. Noponen for these organizations.

Parts of this presentation were adapted from presentations and materials created by Dr. Noponen.

Anna and Katrina are currently working with Dr. Noponen’s family and colleagues to preserve and extend the legacy of ILS. Please consult with us for attribution and usage

information prior to distributing or re-using these materials.

Innovative mind, open heart. HELZI NOPONEN, 1953-2012

helzinoponen.wordpress.com

Muhammad Yunus, Nobel Prize Winner, with Helzi Noponen, 2008. Photo source: Gary Smaby

THANK YOU!

KATRINA MITCHELL Research and Strategy for Good

[email protected]

612-269-2891

ANNA MARTIN Thinking Evaluation

[email protected]

612-805-6460

internallearningsystem.com

ILS | Internal Learning System www.internallearningsystem.com

Evaluation

Some of the evaluation is for impact assessment (proving) perspective, many of them help the participant to assess their own life and circum-stances in a meaningful way. Many have never thought in these terms before. Simply writing down and quantifying thier life and experience can provoke new ways of doing things, new ways of thinking.

• Yes/no questions• Quantity questions• Multiple choice questions• Ranking and/or scaling questions

Planning

Planning tools allow for us to take on greater complexity in our lives than if we try to keep it all in our heads. The planning exercises in the diary open up a whole new world for participants. For many, it is the first time they have purposefully created and acted on a plan. These plans help participants move from reactive/crisis orientation to a proactive/future orientation.

It is important that the participant takes an active role in their own planning. The diary is a tool to identify, prioritize and then strategize about the problems and possibilities in their life before taking action.

Planning components also allow for accountability -- both the participant to themself and the partici-pant to the group, vice versa.

• Problem Sorting Exercises• Goal Setting Exercises• Priority Choosing Exercises• Planning Formats

Learning

Heavily facilitated, typically in a group setting, by the program o!cer or field agent. These exercises make up the “curriculum” of the program; they are messages from the program/agency to its constitu-ents about the intervention. They tend to be problem-solving or strengths-based. The purpose is to inspire participants on how life could be, or how a problem could turn out well. They are not prescriptive but are conversation starters.

• Bad Scene / Good Scenes• Cautionary Cartoons / Illustrated Folk Tales• Dream Scene• Panorama Scenes (e.g. Gender Benders)• 6 panel Stories (photo novella, picture parade,

comic strip)

The diaries are composed of a series of themed modules. The modules together form a development curriculum that is a logical whole. Each module utilizes a variety of di"erent approaches, or components, for meeting program goals and evaluation. The three types of components are for learning, planning and evaluation. Every module does not have all three, but most include at least some evaluation exercises. Usually a learning exercise introduces the module.

Building a Diary | Components

ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell. To reproduce or use please contact us at internallearningsystem.com.

Image source: Helzi Noponen

Image source: Helzi Noponen

Building a Diary | 5 Tasks of ILS


5. Documenting, sharing and reinforcing values

4. Plan or Alter Strategies, Training

3. Analysing Causes of Change / Troubleshoot

2. Assessing Data / Change

1. Collecting Data

The core of the ILS strategy is to engage the participants in a five-step process that carries over and informs how they will think about their life. We all do these things, but the diary makes this process explicit, in order to empower the participants with choice.

Sharing between participants can be a wonderful source of inspiration, support and power; discus-sion is an oft over-looked learning tool.

The diary acts as a testimony to individual experiences, which are often duplicated between group members. These recorded similarities give permission to discuss sensitive wider social issues and inequitable structures and the courage to attempt change. Collective action to summon needed resources and services is bolstered by the diaries’ written record.

With an understanding of the forces at play, participants make intentional changes to their strategies and behavior for achieving their goals. They begin to track and assess data again, to understand the impact of this new plan of action—gaining greater self-awareness, confi-dence and better results in the process.

Once a goal is set, or a pattern emerges, participants use their data and assessment of the situation to identify potential causes and reasons. Asking “why?” and pursuing an answer calls forth the wisdom and skills of each participant, empowering them to step into a position of primary actor in their lives.

Participants, though illiterate, can “read” their own diary and that of others. Individually, and in the context of a group, they gain an understanding of their changing life and livelihood situation. Patterns emerge, the ability to set goals, prioritize problems and solutions and plan for the future are all facilitated through the use of the diary.

Participants own the ILS diary. It is their record. The diary promotes reflection on their current situation, in a struc-tured and relevant format. Each indicator picture has an implied program value attached to it. The process of carefully marking and keeping the diary acts as a green light for them to dare to think about achieving a better life.


The source for all images on this page: Helzi Noponen


PROVING IMPROVING / LEARNING

“Traditional” Impact Assessment Participatory Impact Assessment (ILS, and others)

Program Impact Assessment Objectives

While ILS can help “prove” program e!ectiveness, and can be used for internal accountability, its real strength lies in its ability to support improving. It is the learning and planning components of ILS coupled together with evaluation that make it a powerful tool which is integral to the program design.

1. Marisol Estrella and John Gaventa, “Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review”,as summarized in Anton Simonawitz, “Making impact assessment more participatory” June 20002. Anton Simonawitz, “Making impact assessment more participatory” June 20003. For a discussion of how ILS di!ers from PRA techniques using pictures see Noponen, H. (2002) The Internal Learning System—a tool for micro-finance and livelihoods interventions. Development Bulletin 57: 106–110.

Traditional impact assessment has an external focus (to the donors and policy makers) and a more quantitative and “objective” approach. It tends to be extractive, the information gathered is primarily used outside of the community and little, if any, feedback is provided to the community in terms they can understand. The purpose of the assessment is to prove e!ective-ness and e"ciency rather than to empower the community, participants or program sta! to learn and improve.1

Participatory Impact Assessment (PIA) is internally focused, aimed at improving outcomes through the empowerment of the program sta! and participants. PIA falls along a spectrum of participatory methods used to gather data from extractive formats to fully engaging participants in evaluating their own progress and the program.2 While some focus on the pictorial aspect of ILS as a way to gather information from a low-literacy population, the true strength of ILS is in its abiltity to empower learning and inspire action.3

Why ILS? | Going beyond proving


Participantlearning

Proving to funders +

organizationalleadership

EMPOWERMENTPlanning

Reflection +Assessment

Self advocacy

StrengthenStakeholder

RelationshipsProving to program

participants and sta!

Sta!learning


Building a diary | How to ILS (the basics)

CREATING A DIARY PUTTING IT TO USE

ANALYSIS & REPORTING

Conduct background research to develop appropriate indicators and

determine barriers to success

Draft a simple survey instrument including demographics and key

indicators of program success

From identified barriers and program goals conceptualize learning and

planning components

Pictorialize the survey (Hire an artist to create drawings)

Train 1-2 sta! in facilitation

Train program sta!

Print and mark diaries with ID and date

Distribute census wide

For baseline make first marks in red

Train interviewers!

Data coding and entry

Analysis

Report to all stakeholders

Develop research design

Select sample based on design(panel or cross-sectional)

Semi-extractive side-by-side interview event. Participatory!

Revise*

Likely, issues arise with:

a. Layout of a question on a page causes confusion

b. Individual pictures are not understood at all, or interpreted di!erently across the group

c. A particular question or learning component causes discomfort, stone-walling or conflict from the group

d. Particular subject matter or issues were overlooked

e. One of the choices of a multiple choice question is incorrect or missing

f. Facilitation of a particular learning component should occur di!erently or at a di!erent time.

Create facilitation guide

Debrief / sharing with participants and sta!

FIELD TEST!!

Draft layout and assemble learning, planning and evaluation components

into modules that form a logical whole.

repeatas necessary

ILS is not just about creating a pictorial diary. It is based on well-grounded research design and starts with the creation of a simple survey instrument using indicators derived from thorough background research on the community and the program. Initial research is often done with a participatory approach, though a survey could be designed in collaboration with program sta! who have su"cient field expertise. In both cases it is critical to do at least one field test before implementation.

In order to make a diary that is interesting and relevant to all participants, it should contain very few skip patterns. The survey should be designed with questions that can be answered pictorially such as status or yes/no, simple quantities, multiple choice and scale ratings. Each indicator should relate to a learning component. Do not include survey questions that are only of interest to program sta! or for external proving.


the internal learning system, assessing impact while empowering internal learning

Documents

use ils

ils right

ils diary

ils toolbox

ils book

photo credit

ils change behavior

agency interactwith