quarterly performance report fy 2019, third quarter report

QUARTERLY PERFORMANCE REPORT

FY2019, THIRD QUARTER REPORT

APR - JUN 2019

USAID/UGANDA PERFORMANCE AND

IMPACT EVALUATION FOR LITERACY

ACHIEVEMENT AND RETENTION

ACTIVITY (LARA)

Contract No. AID-617-C-16-00003

This publication was produced for review by the United States Agency for International Development. It was

prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not

necessarily reflect the views of the United States Agency for International Development or the United States

Government.

ID/UGANDA PERFORMANCE AND

IMPACT EVALUATION FOR

LITERACY ACHIEVEMENT AND

RETENTION ACTIVITY (LARA)

QUARTERLY PERFORMANCE

REPORT

FY2019, THIRD QUARTER REPORT

APR- JUN 2019

Prepared under Contract No.: AID-617-C-16-00003

July 30, 2019

Submitted to:

USAID/Uganda

Submitted by:

NORC at the University of Chicago

Attention: Ritu Nayyar-Stone

[email protected]

Phone:+1-301-634-9538

DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the

views of the United States Agency for International Development or the United States

Government.

mailto:[email protected]

Uganda LARA – AID Contract No. AID-617-C-16-00003

QUARTERLY PERFORMANCE REPORT 2019 QPR 2 i

TABLE OF CONTENTS

TABLE OF CONTENTS ............................................................................................................... i

ACRONYMS .................................................................................................................................. ii

INTRODUCTION ......................................................................................................................... 1

IMPACT EVALUATION ACTIVITIES ....................................................................................... 2

UPDATE ON IMPACT EVALUATION ACTIVITIES ..................................................................................... 2 MAJOR ACCOMPLISHMENTS AND CHALLENGES OF PAST QUARTER .......................................... 6

MAJOR ISSUES THAT NEED TO BE RESOLVED .......................................................................................... 6

PERFORMANCE EVALUATION ACTIVITIES ......................................................................... 6

UPDATE ON PERFORMANCE EVALUATION ACTIVITIES ...................................................................... 6 MAJOR ACCOMPLISHMENTS AND CHALLENGES OF PAST QUARTER .......................................... 6

MAJOR ISSUES THAT NEED TO BE RESOLVED .......................................................................................... 6

P&IE PLANNED ACTIVITIES FOR NEXT QUARTER ............................................................ 7

ANNEX A. MIDLINE IMPACT EVALUATION SRGBV TRAINING AGENDA........ 8

ANNEX B. RETENTION AND ATTENDANCE STUDY PRESENTATION ............. 9

ANNEX C. CLASSROOM OBSERVATION STUDY PRESENTATION .................. 17


QUARTERLY PERFORMANCE REPORT 2019 QPR 2 ii

ACRONYMS

CAPI Computer Assisted Personal Interviewing (

CRO Classroom Observations

CSR Centre for Social Research

EGR Early Grade Reading

EGRA Early Grade Reading Assessment

FY Fiscal Year

FGD Focus Group Discussion

LARA Literacy Achievement and Retention Activity

MoES Ministry of Education and Sports

NORC NORC at the University of Chicago

P&IE Performance and Impact Evaluation

RTI Research Triangle Institute

RWI Research World International

R&A Retention and Attendance

SRGBV School-Related Gender-Based Violence

STS School-to-School International

TASO The AIDS Support Organization

USAID United State Agency for International Development


QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 1

INTRODUCTION

NORC at the University of Chicago (NORC), in partnership with subcontractor Panagora

Group, is pleased to submit this Quarterly Performance Report covering the period April to

June 2019 under the USAID/Uganda Performance and Impact Evaluation (P&IE) of the 5-year

Literacy Achievement and Retention Activity (LARA) in Uganda.

LARA is a 5-year (April 7, 2015 to April 6, 2020) USAID-funded initiative to improve reading

skills for 1.3 million learners in 28 districts throughout Uganda. The project, which is being

implemented by Research Triangle Institute (RTI) International, is designed to support the

Ministry of Education and Sports (MoES) - Uganda in its efforts to improve early grade reading

(EGR) and retention. LARA has two main objectives:

Result 1 (R1) focuses on strengthening the capacity of MOES and other educational

stakeholders to deliver EGR. To this end, the activity focuses on improved reading skills

in three local languages (Luganda, Runyoro-Rutooro, and Runyankore-Rukiga) and

English for early primary grade learners P1-4.

Result 2 (R2) focuses on promoting a safer primary school learning environment to

prevent and reduce incidents of school-related gender based violence (SRGBV).

The activity hypothesizes that reducing SRGBV will increase learners' retention because they

will be able to focus on their lessons and feel secure in their learning environment, thereby

improving their ability to learn to read. To realize these objectives, the activity focuses its

efforts on systemic capacity building of the education system, school-level support, and

community- and household-level support and participation.

The Uganda LARA P&IE activity (April 2016 - April 2021) has two objectives: (1) to assess the

impact of LARA on learners’ literacy skills and retention rates; and (2) to assess the

performance of LARA in terms of project management, learning, design, implementation,

results, and sustainability.



IMPACT EVALUATION ACTIVITIES

UPDATE ON IMPACT EVALUATION ACTIVITIES

Preparation and training to collect Midline Impact Evaluation SRGBV data

NORC, in collaboration with Panagora and local firm Centre for Social Research (CSR), will

collect Midline Impact Evaluation SRGBV data in July 2019 in 11 districts. NORC will gather

SRGBV-related data through the following:

Female learner survey

Male learner survey

Primary caregiver survey

Teacher survey

Head teacher survey

Focus Group Discussions (FGDs) with learners

FGDs with primary caregivers

FGDs with teachers

School Safety Observation Tool

Prior to data collection, the LARA P&IE team updated the baseline evaluation instruments to

include questions and get feedback from respondents on LARA implemented activities. NORC

programmed the surveys and observations tool in the NField Computer Assisted Personal

Interviewing (CAPI) application.

In addition, the team updated manuals and the child protection referral forms, and created

introduction letters to allow the fieldwork team access to schools in the districts where data

collection takes place. In addition, CSR conducted advance school visits to gain parental consent

for learner participation in the data collection.

SRGBV expert, Marie-Celine Schulte from Panagora, joined LARA P&IE Evaluation Lead, Dr.

Ritu Nayyar-Stone and Mr. Carlos Fierros to provide training in Kampala between June 19th

and June 28th, 2019. The training agenda is included in Annex A.



CSR was able to retain about 80 percent of the enumerators, moderators, counselors, and

supervisors that had participated in the baseline data collection. This was beneficial in ensuring

continuity in the data collection, counseling, and child protection referral processes. The other

20 percent researchers were recruited on the basis of their qualifications (higher education in

social sciences), experience working with young people and completion of training on

protection of human subjects. CSR also ensured that they had some previous training on

gender-based violence. All new researchers were fairly younger (between 21 and 30 years of

age) so that they could comfortably interview children. Finally, all were confirmed to have

certificates of good conduct from International Police, which qualifies them to work with young

people.

Six of the FGD moderators and note takers from the baseline (2 female and 4 male) were

replaced due to the fact that they delayed submitting their baseline transcripts. The 6 new FGD

experts were cleared after scrutinizing some of the transcripts they prepared recently. They

also have experience working with young people.

All counselors were recruited on the basis of their experience working as counselors with

children. All have certificates in guidance and counseling as well as protection of human

subjects.

Overall, there will be 8 survey teams (4 enumerators, 1 supervisor and 1 counselor). As much

as possible, CSR ensured that there were equal number of male and female researchers

because they will be interviewing male and female respondents respectively. For FGDs, CSR

recruited 4 teams (2 Luganda and 2 Runyankore). Each pair (male and female) will handle FGDs

involving respondents of the same gender. Each of the FGDs will have a same gender counselor

to support the team in case there are children that need their attention.

Preparation for Year 3 Term 2 Retention and Attendance (R&A) data collection

NORC’s local data collection firm partner, Research World International (RWI), will conduct

R&A data collection for year 3 term 2 in 71 schools for a total sample of 3,505 learners

currently in P1 through P7 (most will be in P3 and P6) in July 2019.

In preparation of the data collection, this quarter, NORC reviewed the data collection

instruments and protocols from the last data collection round, and updated them based on

lessons learned from prior fieldwork and data cleaning. NORC’s Nfield programmer updated

the student instrument to show the correct grades in which the pupils are enrolled, and

preloaded the names of the P3 and P6 assigned teachers in the teacher instrument.



Analysis of R&A Data

NORC compiled and harmonized the R&A data from six waves1 of data collection (terms 2 and

3 of 2017; terms 1, 2 and 3 of 2018; and term 1 of 2019) in order to conduct a brief analysis

and produce a set of graphical representations of the data. NORC is including the data displays

in a brief summary report. A few key findings are:

Dropout rates are high, reaching 10.5 percent in Term 1 of 2019. Boys drop out of school

in statistically significant higher proportions than girls.

Absenteeism is high. Average learner attendance is 82 percent each school day. P4 learners

were slightly more likely to attend school than P1 students. Attendance levels are slightly

better in Runyankore/Rukiga than in Luganda dominant regions. Likewise, girls show slightly

better attendance than boys.

Repetition rates are high: around 20 percent of learners failed to pass to the next grade

when they returned to school in 2018, and 36 percent by 2019.

Seventeen percent of the classrooms visited did not have a teacher -assigned or substitute-

present. In some cases, the teacher was in the school but not present in the classroom.

During an average visit day, almost 8 percent of the teachers were absent.

The LARA P&IE Evaluation Team Lead, Dr. Ritu Nayyar-Stone, presented the results of the

analysis of the R&A data to USAID/Uganda on June-26th, 2019. Annex B includes the

presentation slides.

Analysis of 2018 Classroom Observation (CRO) Data

The P&IE Early Grade Reading CRO Expert, Dr. Ursula Hoadley, performed qualitative and

quantitative analysis on the CRO data gathered during the Term 3 of 2018, and wrote a

comprehensive report. A few key findings are:

Greater fidelity to the program structure was evident in 2018 than in 2017. However, all

learners had the LARA books in fewer classes in 2018.

1 Term 1 in 2017 was used to list the target learner population – i.e. the sample – for future retention and

attendance checks.



Classroom discourse patterns had not changed very much and, as in 2017, tended to follow

traditional patterns of highly repetitive chorused readings of syllables, words, sentences or

short texts.

Teachers tended to follow the LARA lesson plans in a very procedural way, rarely

exercising any agency in contextualizing the lesson plans in their classrooms.

The LARA P&IE Evaluation Team Lead, Dr. Ritu Nayyar-Stone, also presented the results of the

analysis of the CRO data to USAID/Uganda on June-26th, 2019. Annex C includes the

presentation slides.

USAID Approval to Drop SRGBV Scenario Approach for Midline Data Collection

NORC had raised concerns about the inclusion of scenarios in the SRGBV assessment tools to

introduce the subject of experiential violence to child respondents and establish a rapport

between enumerator and child. Based on several email discussions and sharing of several other

epidemiological best practice surveys that do not include scenarios, the USAID/Kampala

Mission supported excluding the scenarios for the LARA P&IE midline assessment. USAID

noted that their reasoning was based on the following:

1. NORC presented a credible case that there is the potential for increased harm to

children based on this approach.

2. USAID received written confirmation from NORC that there is no risk to the impact

evaluation's ability to measure LARA's program effects across the treatment arms.

LARA P&IE’s COR, Kay Leherr noted that going forward USAID plans to work with its

partners and stakeholders to continue investigating the most appropriate approaches for

researching child-related violence themes. Based on these ongoing consultations, they reserve

the right to reinstate the scenarios for the LARA P&IE endline assessment should the full

preponderance of evidence point in that direction. Given this go ahead NORC dropped all

scenarios for the pilot conducted on June 27th and field data collection.

Submission of Project Deliverables

Submission of Annual Report/Work Plan (Deliverable #21)

In May 2019, NORC submitted the 2019 Annual Report/Work Plan for AID-617-C-16-00003,

Uganda Performance and Impact Evaluation for Literacy Achievement and Retention Program

(LARA).



MAJOR ACCOMPLISHMENTS AND CHALLENGES OF

PAST QUARTER

The major accomplishments of this past quarter were:

Preparations for SRGBV data collection and enumerator training without major challenges.

The presentation of preliminary findings based on the analysis of R&A and CRO data.

Submission of 2019 Annual Report/Work Plan.

MAJOR ISSUES THAT NEED TO BE RESOLVED

None.

PERFORMANCE EVALUATION ACTIVITIES

UPDATE ON PERFORMANCE EVALUATION ACTIVITIES

The LARA P&IE team and its subcontractor, Panagora Group, prepared an Implementation

Update of the LARA activity throughout Project Years 2 and 3, which also includes a program

summary. The main sources included reports made available by USAID/Uganda, as well as

clarifications provided by Geri Burkholder, LARA Chief of Party, about training schedules, EGR

support supervision, and SRGBV referral systems and trainings.

The Implementation Update was useful during the development of instruments of the 2019

midline impact evaluation, and another update will be done by the end of 2019 to inform the

2020 final performance evaluation. NORC also intends to use midline findings of the impact

evaluation (October 2019) to inform the performance evaluation (February 2020) in such

matters as the locations of schools to select for field visits and the questions to be asked during

the key informant interviews.

MAJOR ACCOMPLISHMENTS AND CHALLENGES OF

PAST QUARTER

None.

MAJOR ISSUES THAT NEED TO BE RESOLVED

None.



P&IE PLANNED ACTIVITIES FOR NEXT

QUARTER

Midline Impact Evaluation SRGBV data collection

NORC’s local data collection firm partner, CSR, will collect data for the SRGBV component of

the Midline Impact Evaluation between July-1st and July-26th, 2019. Survey data collection will

use tablets and the Nfield CAPI application.

Training and fieldwork to collect Year 3 Term 2 R&A data

RWI will conduct R&A data collection for Year 3 Term 2 between July-15th and July-25th, 2019.

Data collection will use tablets and the Nfield CAPI application. In preparation of the R&A data

collection, RWI will hold a training on July-12th, 2019.

Preparations to collect Year 3 CRO data

The CRO Expert, in collaboration with the LARA P&IE team, will adapt CRO instruments and

protocols based on the experience and lessons learned from Year 2 fieldwork and analysis.

Year 3 fieldwork will occur in October 2019 (during Term 3).

During Year 3 Term 2 data collection, RWI will record 30-minute videos of literacy lessons of

six classes (2 treatment P1, 2 treatment P3, 1 control P1, and 1 control P3). Our Expert will

use the video recordings during the training for CRO data collection, expected in late

September/early October 2019.

Preparations and training for fieldwork to collect Midline Impact Evaluation EGRA

and Year 3 Term 3 R&A data

NORC, in collaboration with RWI and School-to-School International (STS), will train

enumerators in the second half of September to collect the EGRA data for the midline impact

evaluation in October 2019. Prior to data collection, NORC will work on the following:

Update the baseline data collection tools to include questions and get feedback from

respondents on LARA implemented activities, and to make the P3 Early Grade Reading

Assessment (EGRA) comparable to the instrument used by LARA.

Get introduction letters that will allow our data collection staff to access schools in the

districts where data collection will take place.

Update training manuals and materials.



ANNEX A. MIDLINE IMPACT EVALUATION

SRGBV TRAINING AGENDA

Time Survey Training FGD Training Counselors

Day 1 (Thursday, June 20)

AM Introduction, Overview and Sampling, The AIDS Support Organization (TASO)

PM Gender and Violence

Day 2 (Friday, June 21)

AM Gender and Violence

PM Child Protection Protocol

Day 3 (Saturday, June 22)

AM Informed Consent/Assent

PM Paper Surveys, Learner & Caregiver Best Practices for FGDs In survey or FGD training

Day 4 (Monday, June 24)

AM Introduction to Tablets Learner FGD Training In survey or FGD training

PM All Group Role Plays, English Caregiver FGD Training In survey or FGD training

Day 5 (Tuesday, June 25)

AM All Group Role Plays, By Language Teacher FGD Training Counselor Training Session

PM Small Group Role Plays Counselor Training Session

Day 6 (Wednesday, June 26)

AM Reliability Tests Counselor Training Session

PM Teacher, Head Teacher, and

School Infrastructure Surveys In survey or FGD training

Day 7 (Thursday, June 27)

AM Pilot

PM Pilot

Day 8 (Friday, June 28)

AM Debrief Debrief Debrief

PM Supervisor Training Supervisor Training



ANNEX B. RETENTION AND ATTENDANCE

STUDY PRESENTATION



ANNEX C. CLASSROOM OBSERVATION

STUDY PRESENTATION

quarterly performance report fy 2019, third quarter report

Documents