Going Digital: Use of Technology in

Monitoring and Evaluation

Urmy Shukla & Sree Sen

CLEAR South Asia at J-PAL South Asia at IFMR

Second Development Marketplace 2013 Convening Workshop

April 24th, 2014

Presentation Overview

Introduction

M&E Data – Challenges and Scope for Using Technology

Using Digital (Mobile) Tools for M&E

When to Use Digital Data Collection

Types of Mobile Data collection – Examples

INTRODUCTION

Abdul Latif Jameel Poverty Action Lab (J-PAL)

Research affiliate network of over 92 professors in 34 universities; 491 evaluations in 56 countries

across 7 global offices

J-PAL South Asia: A Brief History

J-PAL South Asia office set up in

2007 at the Institute for Financial

Management and Research,

Chennai

Over 90 completed and ongoing

projects across South Asia:

education, finance & microfinance,

environment & energy, health,

political economy & governance,

labor markets and agriculture

Scientific Directors: Esther Duflo

(MIT), Iqbal Dhaliwal (Ex-IAS –

1996 batch)

CLEAR Initiative

• Global initiative coordinated by the

World Bank Independent

Evaluation Group

The CLEAR South Asia Regional

Centre has been hosted by J-PAL

South Asia at IFMR since May 2011

Established a partnership with the

Centre for Economic Research

(CERP) in Pakistan and works

closely with Innovations for

Poverty Action (IPA) in Bangladesh

CLEAR: Regional Centers for Learning on Evaluation and Results

CLEAR’s Strategic Vision:

• Building evaluation skills of

practitioners, program

implementers, and donors

• Strengthening M&E systems

and practices for strategic

clients

• Building a culture of evidence

based decision-making and

strengthening demand for

evaluations

CLEAR South Asia Mandate

CLEAR South Asia strategy includes

a focus on the use of cutting-edge

technology for effective collection

and utilization of M&E

In collaboration with Fieldata.Org,

CLEAR South Asia aims to promote

the use of M&E technology, such as

mobile phones and PDAs, for more

real-time aggregation and effective

utilization of M&E.

Context – E-society

Emphasis on the use of

technology for more effective

collection and utilization of M&E

The fundamental objective of all e-Society

initiatives is to make Information,

Communication & Technology (ICT)

more inclusive – i.e. ALL members of

society, especially those that are socially

and economically marginalized, should gain

access to and benefit from the knowledge,

power and opportunities brought about

by new ICTs

About Fieldata.org

Fieldata.Org is a mobile-&-web portal for NGOs to raise funds, by offering

real-time monitoring, and objective data for evaluation.

Mission is to improve transparency & decision-making in development

organizations and government agencies, by empowering them with

technology-tools for better monitoring, sharing, and application of data.

Want organizations and donors to objectively answer:

• Which projects should resources be allocated?

• How well do projects utilize these resources?

About Fieldata.org

M&E DATA

Challenges and Scope for Using Technology

Programme Theory – Log Frame

Inputs/Program Activities

Outputs Intermediate

outcomes Goal

What we do as

a part of the

program -

deliver, teach,

offer loans, etc.

What are the

resources used

–funds, staff,

equipment,

curriculum, all

materials.

Tangible

products or

services

produced as a

result of the

activities -

usually can be

counted.

Short-term

behavioral

changes that

result from the

outputs -

preventive

health habits,

usage of

tablets.

Long-term

changes that

result from

outcomes – the

result of the

program.

Reporting: On expenditures, activities, coverage (targets)

Monitoring : Of use of inputs/activities and process –

Random spot checks of intermediate outcomes: Annual surveys

Evaluation: To assess long-term outcomes and

impact through studies

Reporting

• On Inputs and Outputs (Achievement of Targets)

Monitoring

• Of Processes and Implementation (Doing things right)

Evaluation

• Of Outcomes and Impact (Doing the right thing)

Management and Decision Making (MIS)

• Using relevant and timely information for decision making (reporting and

monitoring for mid term correction; evaluation for planning and scale up)

ALL OF THE ABOVE DEPEND ON THE AVAILABILITY OF RELIABLE, ACCURATE

AND TIMELY DATA

M&E Framework – Use of Data

Problems in Data Collection and Management

What do the following mean?

Data reliability (will we get the same data, when collected again?)

Data validity (Are we measuring what we say we are measuring?)

Data integrity (Is the data free of manipulation?)

Data accuracy/precision (Is the data measuring the “indicator” accurately?)

Data timeliness (Are you getting the data in time?)

Data security/confidentiality (Loss of data / loss of privacy)

Can technology help us address the above issues? How?

USING DIGITAL (MOBILE) TOOLS FOR M&E

Why use technology to collect data?

Digital Tools for Data Collection

Digital Data Collection

Device Data Software Data Transmission Data Visualization

Mobile Devices

Open Data Kit,

Other softwares

Real time Dashboards,

Reports, summary

Tables, Graphs

Laptops,

Computers

Visual Basic, Java,

SQL, etc.

Online / Offline Excel/.csv files,

customized MIS

reports and

dashboards

GPS Arch GIS Real time Maps, Coordinates

Smart Cards and

hand-held devices

Software Online / Offline Summary tables /

customized reports

Biometric devices

(online)

Software Online / Offline

Summary tables /

customized reports

Why use Mobile Technology in Evaluations

Improving transparency & accountability in development organizations and

government agencies, though technology-enabled M&E for better

monitoring, sharing, and application of data.

Enabling organizations, donors and citizens to use M&E data for real-time

decision-making, better implementation and delivery of projects and

services

Mobile Technology Options

How is Mobile Technology Used

Why Mobile Data Collection?

Real-time data from the point of collection

Built-in logical flow and validation checks improves data quality

Ability to collect new types of data – Location (GIS), media (pictures,

audio)

Cost effective over time- involves one-time hardware costs and ongoing

maintenance. No paper, printing costs

Easy to manage and analyze large amounts of data

Reduces intermediate levels of data transmission

Why Digital Technology for Routine Monitoring

Paper Reports

Delay in reporting

Multiple levels of reporting

Information flow is one way

and not actionable

Bulky hard copies of reports

Errors in entry, needs

additional scrutiny

Using Mobile Phones

Almost instantaneous

reporting

Direct reporting

Information flow is both ways

and actionable

Web-enabled reports

Field-level checks on quality

incorporated

Why Digital Technology for Evaluations / Periodic Monitoring

Paper Surveys

Printing, tracking forms is tedious

for large surveys, changes costly

Many errors in data collection,

entry, manual scrutiny

Effective monitoring of data quality

is complicated and laborious

Requires additional hardware

devices for non-text data such as

gps, pictures, audio etc.

Using Mobile Phones

Deployed remotely, tracked in real

time, changes easy on the field

Limited errors due to built in logic

flows and validation, Single entry

Real time tracking, time/date/GPS

features improve monitoring

Single device for gps, audio,

pictures. Easy to integrate, can be

used for verification

Limited flexibility on field: Standardized formats for data collection; upfront

development of software tools upfront

Local language compatibility: Being developed for all languages

Typing may be slower: learning may take time (scribbling on paper is faster)

Lack of connectivity: on field limits real-time transmission of data

Potential Challenges with Digital Data Collection

TYPES OF MOBILE DATA COLLECTION

Examples

Data Quality & Real-time Data for Monitoring Purposes

Improving:

• Data quality

• Speed

• Transparency

• Accessibility

• Flexibility

Quick & Easy Set up:

Mobile-based Monitoring and Evaluation in Action:

MFI agents entering weekly loan repayments

for instant tabulation

Community health workers feeding

back information on beneficiaries for

automatic identification of high risk

cases

Auditors collecting survey, observational,

photographic and GIS data on

infrastructure in slums.

Use of Mobile-based Technology in M&E

REPORTING/MIS

Routine (Real-time) reporting

• Weekly loan repayment information of MFI clients reporting by field staff

MONITORING

Ongoing program monitoring

• Beneficiary information collected and sent by health workers (for disease surveillance, delivery of benefits etc.)

Occasional (Surprise) checks

• Spot checks by supervisors to monitor attendance and performance of staff

EVALUATION

Survey data

• Household survey data to assess impact of <<xx>> program

Use of Mobile Technology as key intervention in Programs

Information/Messages

• SMS reminders to beneficiaries about important health activities

(treatment compliance)

Implementation Tools

• Videos and online teaching aids used in schools for regular or remedial

education programs

Biometric, GIS information

• Record beneficiary information for easier tracking and follow up.

Complementarities with other programs by same organization

Case Study: Delhi Voters Project

Project description and objectives

• An RCT evaluating whether providing information to government

officials and slum dwellers can lead to higher accountability and

thereby improved service delivery.

Interventions evaluated:

• The effect of providing information on spending and quality of public

services delivered by elected officials during election sensitive periods

Role of mobile-based data collection

The intervention

• Field-based audits of public services in slums using mobile-based

technology

• Surveys and observations

• Photographs

• GIS indicators

• Send report cards to elected representatives highlighting the quality

and access to public services in their area

The Implementation

GARBAGE FACILITIES

1) Dumpsters 2) Bins

3) Informal Points

TOILETS

Open Public Toilet

Complex

Public Urinals

Started by auditing …

Audit Report Cards

Going a step further with Mobile Based GIS software

Accessibility to public services

• How far is the nearest

toilet?

• How far is the nearest

formal point of

disposing garbage?

Spread Analysis

• Does cleanliness of a

toilet affect household

health?

MOBILE GIS !

WHEN TO USE DIGITAL DATA COLLECTION?

When to Use Digital Technology?

Mobile Phones / Tablets

(Smart phones)

SMS/IVRS based (Simple

phones)

GPS / Biometric Devices

Smart (Biometric) Cards

For periodic surveys, mapping,

supervision, electronic databases

Routine monitoring for delivery of key

services

Mapping and data verification at a micro

level (HH, beneficiary)

Direct payment to beneficiaries upon

verification, create digital databases

Initial one-time cost of devices

Additional costs for

maintenance (batteries,

replacement etc)

Ongoing data-plan costs, and

service-provider costs

Real-time access to data to

monitor quality and progress

Environmentally friendly

Mobile

No one-time ‘hardware’ cost

Ongoing costs of printing,

transporting and storing paper

questionnaires

Data-entry operations take

significant time and resources –

training

Longer time-frame before data

is available for analysis

Paper

Cost Effectiveness

THANK YOU