Appendices to Terms of Reference –evaluation of Nal’ibali reading clubs

Appendix A – background information for

research team

Background

The Nal’ibali campaign In 2012, Nal’ibali (“here’s the story” in isiXhosa) launched a national reading-for-enjoyment

campaign. It seeks to give all children in South Africa opportunities to fall in love with books and

stories, by creating the conditions that support children to become readers: access to reading

material; role models who model and encourage reading; frequent opportunities to read, write and

share stories; and awareness that reading for joy supports educational achievement.

“The Nal’ibali way”: reading for enjoyment Literacy skills are a strong predictor of future academic success in all subjects. Children who regularly

read and hear engaging stories, in languages they understand, are well equipped and motivated to

learn to read and write. A significant body of research reinforces links between access to high-quality

reading materials, motivation/interest in reading and regularly reading for pleasure, and improved

outcomes for children. Similarly, research demonstrates that children who learn to read well in their

home language do better in school in both in their home language and in additional languages.

The Nal’ibali model

Training curriculum Nal’ibali delivers its training in four modules:

(1) Introduction to Nal’ibali

(2) Be a FUNda Leader

(3) Run a reading club

(4) Training for trainers

For more information about the training curricula, objectives and content, please see

https://nalibali.org/training-options.

It has also developed customised training curricula for special projects. For example, the Story

Powered Schools project, which is working in 720 Eastern Cape and KwaZulu-Natal primary schools

over 3 years, offers a two-day training for educators, and a one-day Community Training for parents

and community members.

Online training modules were developed in 2018 and are being launched in 2019.

Partnership model Nal’ibali invites partner organisations – which may be ECD forums, schools, non-governmental

organisations (NGOs), government departments, faith-based communities – to sign up for a six-

month training and mentorship package. Most partners complete the first three training modules

outlined above, with practical tasks to complete in between each module. Nal’ibali supports

partners to register and run reading clubs, and invites them to a graduation “imbizo” where they can

connect with other partners working in their geographical area.

This package is primarily offered to organisations in the areas where Nal’ibali’s field staff are working

on the ground. Our team is concentrated in our four priority provinces (those with the highest

number and population density of children): Eastern Cape, Gauteng, KwaZulu-Natal and Limpopo.

General public Nal’ibali also invites all interested people in the general public to be reading activists, which Nal’ibali

calls “FUNda Leaders.” FUNda Leaders are people who read aloud to and share stories with the

children in their lives; join their local libraries; start reading clubs; and advocate for the power of

reading with their friends, family and community. Nal’ibali’s website, www.nalibali.org, offers a

wealth of free resources – including multilingual stories, audio stories, tip sheets, activities, Story

Power guides, printable story cards and Nal’ibali reading supplements. FUNda Leaders can also

interact with Nal’ibali via social media.

Nal’ibali supplements are also available to the general public inside select Tiso Blackstar

publications, including the Sowetan (Gauteng, North West, Limpopo, KwaZulu-Natal, Free State), the

Sunday Times Express (Western Cape), the Daily Dispatch (Eastern Cape) and the Herald (Eastern

Cape).

Members of the general public can also request training for their organisations or communities.

Research to date A summary of Nal’ibali research undertaken to date is available in this document:

Research at Nal’ibali: 2017-18 overview and towards a 2019+ agenda (date: Sept 2018).

Additionally, Nal’ibali recently received feedback on an external evaluation of its Story Powered

Schools programme. This is not included in the linked document and is summarised below.

Story Powered Schools external evaluation An external evaluation of Nal’ibali’s Story Powered Schools project, commissioned by the United

States Agency for International Development (USAID), used a randomised controlled trial (RCT) to

evaluate the programme’s impact on children’s reading outcomes.1 It is also looking at teachers’

attitudes towards reading, reading behaviour and classroom practices; children’s attitudes towards

reading and reading habits; and access to/use of reading materials in schools and homes.

The study is still underway, but the endline for the first cohort (of two) is complete. Although the

study identified some small, positive changes in the enabling conditions at treatment schools when

1 For more information about the evaluation design, please see the summary in this document: Research at

Nal’ibali: 2017-18 overview and towards a 2019+ agenda

compared to control schools, it did not find any impact on the ultimate outcomes (children’s reading

skills) or intermediate outcomes (children’s attitudes and behaviours; teachers’ attitudes and

behaviours). The endline was conducted almost a year after the project concluded in most schools,

which may have influenced these findings.

A qualitative component to the study used observational research to examine how time is used in

schools and the nature of activities conducted by Nal’ibali staff. This research has not yet been

finalised, but at a high level: these case studies, while only at a few schools, noted that:

- Story Sparkers struggled to get opportunities to work with children, and sometimes used

teaching and learning time to run reading for enjoyment activities.

- Although Nal’ibali hoped teachers and community members would run reading clubs,

most reading clubs were run by Story Sparkers alone. More than half took place in

conventional classroom settings with large groups of children.

- As a result, Story Sparkers chose activities and strategies that allowed them to manage

large groups of children. Sometimes they struggled to manage behaviour; to structure

and pace activities; to support learners and give feedback to them.

- Only about a quarter of time observed was spent engaging with text or writing. Only 7%

of this time was spent where Story Sparkers read aloud to children; much of the time,

“reading” was children reading aloud. They often struggled and their peers lost interest.

A more scripted, structured day for Story Sparkers is recommended. The implication for

this research is that more structure may also be useful for reading club leaders.

Relevant research conducted by others In addition to Nal’ibali research, a number of other research

Indicators of quality at reading clubs

Data analysis report More information about Nal’ibali’s reading club network, including analysis of dimensions such as

child-to-adult ratio and how it varies by the type of setting (school, educare, library, home etc) is

available in the Nal’ibali 2018 year-end data analysis report.

Training curriculum Nal’ibali’s reading club training curriculum will be provided to the successful research team.

Story Power Guide A guide to running reading clubs is included as Appendix C in this document.

Draft QA form In 2019, Nal’ibali drafted and is piloting a reading activity observation tool. This tool was developed

by Nal’ibali’s Research & Innovation team with input from trainers. It is included as Appendix D

below, to provide more granular information on what types of literacy development practices

Nal’ibali looks for at reading clubs.

Child protection policy Nal’ibali’s child protection policy is available on our website and can be downloaded here.

Appendix B – Nal’ibali research principles

As per Nal’ibali’s 2019-2021 research agenda2, the following principles should guide its research agenda and activities:

● Aligned and adaptive: Nal’ibali’s research agenda should be aligned to the organisational strategy. It should be dynamic, with space to adapt as we identify new research questions and learn new things.

● Balanced: Nal’ibali’s research should include a mix of formative (process) research, summative (outcomes/impact) research, and prototyping/testing. It should use both qualitative and quantitative methods. It should include internal evaluations and external research.

● Useful and learning-focused: Research should be utilisation-focused3 – designed to provide insights that are useful and relevant to campaign design, implementation and advocacy.

● Relevant and influential: Nal’ibali should conduct some research with relevance to the literacy development sector more broadly, and package and communicate it in ways that can inform and influence practitioners and policy makers.4 It should also share what it is learning regularly at conferences and in relevant forums, and seek to publish peer-reviewed research, and include this goal in the planning process, where appropriate.

● Collaborative: Nal’ibali should work with universities, other researchers, literacy development organisations and behaviour change campaigns to design and conduct research, and to share tools and findings.

● Participatory: Internal research should be designed and conducted in a participatory manner that addresses learning needs across departments and builds team members’ skills and capacity. Each major/external research project should have a steering committee of staff members with relevant, diverse skills and insight. Research dissemination strategies should ensure we create opportunities for staff to learn from research conducted.

● Accountable and accessible: In its research, Nal’ibali should be accountable to stakeholders and communities where we work; to staff members; to partner organisations; to the board; and to funders.Research should be should be shared widely with all stakeholders, in a format that is easy to understand, and should be accessible to the general public.

● Rigorous: While Nal’ibali should use the appropriate degree of rigour for each project,5 it should employ random samples wherever possible to investigate key questions about our network and verify internal evaluation data, so findings are generalisable.

2 Available here. 3 See https://www.betterevaluation.org/en/plan/approach/utilization_focused_evaluation for more

information about utilisation-focused evaluation. 4 For example, design and position research in relation to the binding constraints to reading; to EGRS and

other DBE initiatives; to consolidate common indicators that can be used to measure reading habits and behaviours. 5 For example, an internal reading club survey does not require the same statistical power or methodological

rigour as an external randomised controlled trial.

Appendix C – Story Power Guide: Reading clubs

Nal’ibali has produced a series of “Story Power Guides”, to provide guidance, tips and ideas for

people who want to do various reading promotion activities.

“Story Power: a guide to reading clubs” can be downloaded here:

https://nalibali.org/sites/default/files/media/story-power-guides/lrez_eng_nalibali_sp_guide_2.pdf

For more information about Nal’ibali’s approach to promoting reading for enjoyment in homes,

schools and communities, all of its Story Power Guides, in all South African languages, can be

downloaded here: http://nalibali.mobi/story-power-guides

Appendix D – draft reading club quality assurance tool

The tool below is currently being pilot as an internal quality assurance tool at Nal’ibali. It outlines

some of the key characteristics and activities Nal’ibali would look for in a reading club.

The form does not include norms or targets. In a few instances, targets for quality are established

within the organisation, such as:

- Child to adult ratio: Nal’ibali intends to have a child-to-adult ratio of between 10-to-1

and 15-to-1 in reading clubs.

- Duration of reading club: To cover the full range of activities, a reading club should

ideally have at least an hour.

Further standards for quality could be explored together with the research team.

QA visits: Reading activity observation Guide

School name: _____________________________________________________

District: _________________________________________________________

Observation Date: _________________________________________________

Cohort: □ 2017 □ 2018 □ 2019

RSP Focused Support School? □ Yes □ No

Grade/s Observed: _______________________________________________

Activity leader Name & Surname: ___________________________________

Activity leader designation:

□ Teacher □ HOD □ Learner □ Story Sparker □ Other: _______________

Any other volunteers? (if yes, how many and role(s)): Yes ___No ___#_______

1.____________________________Role______________________________

2.____________________________Role______________________________

Facilitator Name & Surname: _______________________________________

Co-facilitator name & Surname: _____________________________________

CONSENT

This school visit is part of internal quality assurance processes

conducted by Nal’ibali.

The purpose of the visit is not to judge or evaluate the activity

or the people involved. The purpose is to understand how

schools are responding to the Story Powered Schools

programme and how it is working in schools.

Your participation in this observation is voluntary. You can

choose to withdraw at any time.

SIGNATURE OF ACTIVITY LEADER:

Required for non-Nal’ibali staff only

Background and instructions

Purpose of tool: This is an observation guide to be used during a school observation visit. The observation process will help Nal’ibali and the project to understand implementation process, identify the outcomes of the project and draw lessons to improve the project. Before you observe: You might need to meet the principal or Big 5 members. Please introduce yourself and the co-facilitator and outline the purpose of the observation process. Before observing the reading activities which are led by the activity leader (Nal’ibali staff or School person), request them to sign the consent section. The provided consent section is to ascertain that the activity leader understand why he/she is participating in this process. This is also to acknowledge their right to participate or withdraw from the process. While you observe: As an observer, find a spot which will not draw attention to you, but to allow the implementation process to take place as planned without your influence. Don’t assist on anything unless it’s for safety precautions. Instructions for observation tool:

1. Review the tool before your visit / observation, so you can remember what to pay attention to while you are observing the activity.

2. Use a different form for each activity you observe. For example, if you spend time with Grade 1s, Grade 2s and Grade 5s, please complete 3 activity observation forms.

3. Complete Parts 1 and 2 of the instrument while you wait for the activity to begin. 4. Complete Part 3 by writing a detailed narrative report of the activity. 5. Remember to take a photograph of any text used in the activity (see 3.2). 6. Complete Part 4 outside the room where the activity took place, working together with the

second fieldworker. 7. If you cannot complete Part 4 at school, because multiple activities take place back-to-back,

complete it as soon as possible after the visit (same day).

Additional tips:

Your documentation of your responses should be based on your experience and your understanding

of the implementation process. Be mindful of your biases, and respond with honesty, objectivity and

openness.

Other key things to observe: Communication/relations: your encounter with the principal, Big 5 representatives and any other elements (e.g. child protection, corporal punishment, etc) that could either hinder or influence the outcomes of the project.

Part 1: Activity structure

1.1 What kind of reading /story activity was observed? Tick line

A Drop Everything And Read (DEAR)

B Before school assembly activity

C Reading club

D Other, specify:

1.2 How many children were involved in the activity? Number

A Number of children

1.3 How many adults were involved in the activity? Number

A Number of adults

1.4 How long did the activity last? “Activity” refers to the whole session (reading club, DEAR period, etc). It does not refer to each individual activity during the reading club.

Enter time/ duration

A Time activity begins:

B Time activity ends:

C Total activity duration:

1.5 In what grade/s were the learners who participated in the activity? Grades:

1.6 Where did the activity take place?

A Classroom

B Library

C Dedicated space for Nal’ibali activities

D Other, please specify:

1.7 When did the activity take place? Tick appropriate line

A Before school

B During assembly

C During lesson / class time

D At break

E After school

F Other, specify:

1.8 Activity leader used the following language/s

All of the time

Half or more than half the time

Less than half the time

Never

A isiXhosa

B isiZulu

C English

D Other, specify

1.9 Children used the following language/s All of the time

Half or more than half the time

Less than half the time

Never

A isiXhosa

B isiZulu

C English

D Other, specify

Part 2: Environment and access to reading material

2.1 Which of the following are in the environment? Tick all that you observe

Tick

A Hanging library

B Storybooks

C Supplements

D Nal’ibali posters

E Children’s writing / own books

F Children’s drawings

G Children’s arts and crafts

H Posters about how to handle/use books

I Reading club code of conduct

J We Love To Read reading club poster (or other record of reading club activities)

K Reading club file / portfolio of evidence

L Other, please specify:

2.2 Which languages are printed materials?

All materials

Most materials

A few materials

No materials

A isiXhosa

B isiZulu

C English

D Other, specify

2.3 To what degree is there a variety of reading material (language, topic, level of difficulty, fiction/nonfiction, etc.) for children to choose from?

A □ Wide variety of materials □ Some variety of materials □ Not much choice

B Discuss:

2.4 Is print-rich content meaningful and relevant? Does it show evidence of writing for real reasons? Is it linked to books and stories children have read? Is it copying out texts, or have children written their own ideas?

A □ Yes, very much □ Yes, somewhat □ No Choose one

B Discuss:

2.5 Is the environment conducive to reading for enjoyment activities?

Does each child have a place to sit (seat / desk)? Is it overcrowded? Are there comfortable places for children to sit and read – e.g. carpets, cushions? Is the set up different from a traditional classroom setting (i.e. desks rearranged)?

A □ Yes, very much □ Yes, somewhat □ No Choose one

B Discuss:

2.6 Is the environment safe from a child protection point of view? E.g. Children have adult supervision; safe sanitation is available; building is fenced; etc

Tick

A □ Yes, very much □ Yes, somewhat □ No Choose one

B Discuss:

Part 3: Activity narrative

3.1 Complete a record of the activity in the table below. Record at least 5-minute intervals

Time What the activity leader and learners do

Time What the activity leader and learners do

Time What the activity leader and learners do

3.2 Take a photo/s of any text used in the activity Tick

A Photos taken

B N/A no materials used or produced in the activity

Part 4: Post-activity observations

4.1 Activities

4.1.1 Indicate which activities you observed Tick the appropriate line

A Adult read aloud to children

B Adult told a story to children

C Children read on their own (independent reading)

D Children read in pairs / small groups

E Children read aloud

F Adult wrote

G Children wrote

H Children did drawing

I Songs and games

J Arts and crafts

K Using Nal’ibali supplement

L Acting out stories (drama)

M Other, please specify

Any comments:

4.2 Text use

4.2.1 Indicate the dominant text used in the course of the activity If more than one text is used, select all and note the MAIN text used

Tick the appropriate line

A Storybook – SPS

B Storybook – Other

C Supplement – cut out and keep

D Supplement – story corner story

E Supplement – Nal’ibali fun

F Supplement - other

G Reader (including ‘big books’)

H DBE workbook

I Textbook

J Text on blackboard

K Loose photocopied worksheets

L Other, specify: ____________________________________________________

M N/A no text used

If a book, cut-out-and-keep book, story corner story or reader was used, please indicate title of the book/story used:

4.2.2 Did children handle any texts themselves? This refers to the different types of texts listed above

Tick the appropriate line

A Yes – all learners got to hold/use a text

B Yes – some learners got to hold/use a text

C Yes – a few learners got to hold/use a text

D No – learners did not handle any texts, only adults used texts

E No – learners did not handle any texts, no text was used

If yes, please describe which texts were used by children and how they were used: Description:

4.2.3 Did children check out books / take any texts home? This refers to the different types of texts listed above

Tick the appropriate line

A Yes – all learners got to check out/take home a text

B Yes – some learners got to check out/take home a text

C Yes – a few learners got to check out/take home a text

D No – learners did not take home any texts

E Other, please specify:

If yes, please describe what happened:

4.3 Reading for enjoyment activities

4.3.1 Did the activity leader use play games / sing songs with the children, in a way that created a relaxed, free environment?

Tick appropriate line

A Yes

B No

4.3.2 If reading a book or story, did the activity leader read out the name of the author, illustrator and / or translator?

Tick appropriate line

A Yes

B No

C N/a no reading in the activity

4.3.3 If reading a book or story, did the activity leader invite children to predict or guess what the story is about?

Tick appropriate line

A Yes

B No

C N/a no reading in the activity

4.3.4 When the activity leader read or told a story did she show her own enjoyment, excitement or pleasure in reading or telling a story?

Tick appropriate line

A To a large extent

B To some extent

C A little bit

D Not at all

E N/a no reading aloud or storytelling by activity leader

4.3.5 When the activity leader read or told a story did she do so with excitement, intonation and/or expression (e.g. to create a sense of anticipation)?

Tick appropriate line

A To a large extent

B To some extent

C A little bit

D Not at all

E N/a no reading aloud or storytelling by activity leader

4.3.6 Did the activity leader indicate any purpose for reading? (e.g. indicate what might come out of reading – learning something new; understanding something; enjoying a story)?

Tick appropriate line

A To a large extent

B To some extent

C Hardly at all

D Not at all

E N/a no reading in the activity

4.3.7 If the activity leader read or told a story, were learners asked to …

To a large extent

To some extent

Hardly at all

Not observed

A Retell, act out or summarise the story they have read or has been read to them

B Make predictions based on events in the story or pictures related to the story

C Identify or restate the main theme/idea of the story

D Answer factual questions from the text or story

E Answer inferential questions about what they have read or what has been read or told to them (e.g. ‘Do you think she was right to…?’; ‘What would have happened if...’?)

F Relate the story to their own experiences and lives (‘Have you ever…’ ‘How would you feel if…’, ‘Do you know anyone who…’)

G Discuss the meaning of vocabulary words in the story

4.3.8 Was the story (either told or read) completed? Tick appropriate line

A Yes

B No

C N/a no reading / storytelling in the activity

4.3.9 Did children have an opportunity to choose books / other texts to read / look at? Were they able to decide what they wanted to read or look at?

Tick appropriate line

A Yes

B No

C N/A – children did not read or handle texts themselves

4.3.10 Did the activity leader write down children’s ideas / thoughts / stories / songs? (Demonstrating the connection between spoken and written language)

Tick appropriate line

A Yes

B No

C If yes, briefly describe what occurred.

4.3.11 Did the children do any written extension activity based on the reading or story? (for example, write a diary entry or letter to a character, design a cartoon strip or new cover for the book)

Tick appropriate line

A Yes

B No

C n/a – no reading or storytelling in activity

If yes, please describe:

4.3.12 Describe the nature of the interaction between activity leader and children. For example, is it nurturing? Does the leader speak harshly to children? Does the leader give feedback to children on their work? Does the leader struggle to control a large group of children? Does the leader role model reading and writing by talking about books they are reading?

Response:

4.4 Reading club information

Complete this section ONLY when you have observed a reading club

4.4.1 What is the name of the reading club? Ask the activity leader if this is not clear

A Name:

4.4.2 Do children choose to participate in this reading club (voluntary)? Ask the activity leader if this is not clear

Tick appropriate line

A Yes

B No

C Don’t know

4.4.3 How often does this reading club meet? Ask the activity leader if this is not clear

Tick appropriate line

A 5 times a week or more

B 2-3 times a week

C Once a week

D Every two weeks

E Other, please specify:

4.4.4 Who usually runs this reading club? Ask the activity leader if this is not clear

Tick appropriate line

A Story Sparker

B Teacher

C Community member

D Other, please specify:

E Don’t know

4.4.5 Has this person been to Nal’ibali training? Ask the activity leader if this is not clear

Tick appropriate line

A Yes

B No

C Don’t know

4.4.6 How long has this reading club been active? Ask the activity leader if this is not clear

Response: