rca introductory note 2014

8/11/2019 RCA Introductory Note 2014

1/22

iWWW.REALITY-CHECK-APPROACH.COM WWW.EDGROUP.COM.AU

in tou ch and up-to-date with the real i t ies of people l iv ing in po verty

A Short Introduction to RCA2014 Edition


2/22

1


3/22

2


4/22

3

TABLE OF CONTENTS

1. OVERVIEW OF THE REALITY CHECK APPROACH ................................................................... 4

1.1. CONTEXT ..................................................................................................................................... 41.2. WHAT IS RCA? ............................................................................................................................... 4

1.3. SITUATING THE RCAWITH OTHER METHODS.................................................................................... 5

1.4. INTENDED OUTCOMES..................................................................................................................... 6

1.5. LIMITATIONS................................................................................................................................... 7

2. CONTEMPORARY RELEVANCE ................................................................................................... 8

2.1. AID EFFECTIVENESS ........................................................................................................................ 8

2.2. VALUE FOR MONEY......................................................................................................................... 9

2.3. OWNERSHI;WHOSE VOICES COUNT? ............................................................................................... 10

3. RCA STUDY DESIGN & IMPLEMENTATION .............................................................................. 11

3.1. CONVERSATIONS........................................................................................................................... 11

3.2. AREAS OF ENQUIRY...................................................................................................................... 12

3.3. THETEAM..................................................................................................................................... 12

3.4. RCAMAIN STUDY......................................................................................................................... 13

4.. NOTES FOR RCASTUDY COMMISSIONERS........................................................................... 17

5. FROM RESEARCH TO ACTION .................................................................................................. 17

6. THE RCACOMMUNITY ......................................................................................... 18

7. FUTURE USE .............................................................................................................................. 19

Acknowledgments

Special acknowledgment is made of Helena Thorfinn, Esse Nilsson and Dee Jupp who were the

inspiration and driving force for the first RCA in Bangladesh in 2007.

Most importantly we acknowledge the many host households for welcoming RCA researchers into

their homes and to their neighbours and local service providers for embracing this approach and

teaching us so much.

Photos on the front cover depict RCA researchers in

Indonesia, Mozambique and Nepal informally

interacting with members of their host households.


5/22

4

1. OVERVIEW OF THE REALITY CHECK APPROACH

1.1. CONTEXT

Would the poor express amazement that people who are experts inpoverty don't even bother to spend

time with them. - Koy Thompson, Director,Action Aid

The context, realities and aspirations of people living in

poverty are changing at accelerating rates. Development

interventions need to keep pace and provide the right

response to peoples emerging needs and ambitions.

Simultaneously, donors are demanding increasingly

responsive and effective development programme and

service provision. The challenge has never been greater for

development professionals and development interventions to

keep up to date and in touch with increasing complexity.

Acknowledging the current demand for evidence based

research (usually referring to quantitative or quasi-

experimental design) to shape policy and practice,the Reality

Check Approach (RCA) offers a powerful complement to

these approaches by providing a different dimension of

evidence. As an approach that is open to multiple and

unexpected realities, RCA can flag up new areas for

quantitative research, offset and reduce its reductionism,

provide detailed understanding of behaviours and insights

into whether and why development interventions are taken up

or not. Policy-makers can be informed of rapidly changing

realities of which they may not be aware and can be helpedto understand how context and behaviour affects the success

of programmes.

Since first being piloted in Bangladesh in 2007 the RCA has

been used in Indonesia, Bangladesh, Nepal, Ghana,

Nicaragua, Pakistan and Mozambique to provide insights for

Government and NGO development programmes supported by Australian Aid, Sida, the EC and DFID.

These insights have been used at national development, programme and project levels.

This short introductory guide provides an overview of critical components and thinking surrounding

RCA for researchers or research commissioners considering including RCA in their suite of research

methods. Practitioners are referred to the more detailed Rough Guide to RCA to be published in 2014.

1.2. WHAT IS RCA?

The RCA is an experimental qualitative approach to collecting and using information from the ground.

The core of RCA is immers ions during which researchers stay several days and nights with people

living in poverty to engage with, listen to, observe and document their voices, opinions and experiences.

Like other qualitative approaches, this process focuses on asking howand why rather thanwhat,

when and how many. The main idea is to have sustained, detailed conversations and intense

interactions with a small number of households in their own homes. Sharing in their lives provides

opportunities to better understand and contextualise peoples opinions, experiences and perspectives.

RCA is a qualitative approach to

feedback and evaluation which

involves outsiders living with people

living in poverty in their own homesand joining in their every day lives.

The relaxed environment this

provides enables easy informal

conversations with all members of

the family , their neighbours and

others who interact with the

household. It allows the outsider to

experience and observe the realities

of the family. This provides a

meaningful basis for joint reflection

on change with the family. It helps

to shed light on the disconnects

between knowledge, attitudes and

practice which conventional

evaluation is often unable to do

RCA IN A

NUTSHELL


6/22

5

The RCA is generally intended to track changes in how

people live and experience their lives and involves

repeating the RCA with the same households at

approximately the same time each year over a period of

several years. Longitudinal studies have the advantage

of engaging regularly with households and communitiesenabling the researcher to build trust and increased

openness in interactions over time which other methods

may lack.

A one-time RCA can also be used as a pulse taker (e.g.

finding out how new programmes are being received,

updating on contextual change) or as a retrospective

study on how people experienced and assessed change

over the past.

1.3. SITUATINGTHERCAWITHOTHERMETHODS

RCA is a hybrid approach that draws on a variety of traditions and qualitative methods, including;

ethnographic studies, people centred research, participatory learning and action, appreciative enquiry,

listening studies, observation, story-telling and participatory visual documentation and evaluation.

Perhaps the most influential method behind the RCA is participant observation and RCA has been

described as a light touch participant observation (though not everyone likes this description).

Participant observation involves both participating in and observing peoples everyday activities. This

process usually entails extensive and detailed research into behaviour, along with exploration of

peoples perceptions and actions over long periods of time. The RCA is similar in that it requi res

researchers to participate in everyday life within people's own environment but differs by being

comparatively quick and placing more emphasis on informal, relaxed and insightful conversations rather

than observing behaviour and the complexities of relationships.

Although RCAs may be stand alone studies, increasingly they

are being used in mixed method impact evaluations as a

means to provide a rich contextual basis for quantitative data

collection. Sequenced carefully, it can help develop better

household survey instruments, provide insights for survey

enumerators on appropriate engagement, provide early

indication of mis-understood or mis-communicated survey

questions, assist in interpretation of data, provide insights into

outliers and their significance, question generalisation and

over-simplification.

Further, there are some issues which do not lend themselves

to surveys or other forms of quantitative data collection e.g.

sexual behaviour, domestic violence, intra-household decision

making and these are better explored in non-threatening,

informal situations of trust which RCA is especially appropriate

for. The longitudinal nature of RCAs and the nuanced data

they collect lends RCAs particularly well to exploring change

processes. There are also small or marginalised groups which

may be left out of statistically driven surveys but whose experience and understanding of programmes

are important. Here again small scale purposive RCAs can fill an important void.

Chatting with Secondary School stud ents on a

beach in Indo nesia- 2010

Qualitative research without

quantitative can be insightful

but vague.

..quantitative without

qualitative can be precise but

may be wrong headed,

misdirected and contrived

Kalinowski, Lai, Fidler and Cumming

MIXED METHODS?


7/22

6

1.4. INTENDED OUTCOMES

A key intention of RCA is to understand better whether and how development policies translate into

effective programmes and change for people living in poverty and how these efforts and changes are

perceived. The RCA provides an opportunity to get closer to these realities and to link them with policy

and practice within the context, limitations and resource constraints of typical development aidprogrammes. It provides valid, up to date, people-centred information and is particularly valuable as a

means to provide explanations of quantitative data by giving time for people to examine and explain

why they make choices, why they take action and what prevents them from exercising choice and

agency.

RCA studies to date have;

Provided immediate insights into new programme activities, e.g. introduction of the new primary

school examination in Bangladesh, distribution of mosquito nets in Ghana

Flagged up issues for further study e.g. increasing consumption of salt in Bangladesh,

increasing drop out of boys from school in Bangladesh and Indonesia

Identified implementation problems in ongoing programmes e.g. manipulation of national

school exam conditions in Indonesia, miss-match between equipment and staffing in health

facilities in Bangladesh, leakage of free medicines to the non-ill in Nepal and Bangladesh, real

costs of education for families in Indonesia (including assertive demands for pocket money)

Identified un-met needs e.g. agricultural extension advice for cardamom farmers in Nepal,

family planning advice for men and unmarried persons in Bangladesh, safe avenues to

complain about government services in Ghana, medicine provision for people living in poverty

in Bangladesh for non-communicable diseases e.g. high blood pressure, diabetes, stress.

Provided a wider lens for evaluation by embracing negative, unintended consequences of

interventions/change and multiple realities e.g. exposed the burden of livestock care for asset

transfer recipients in Rwanda, increase in snack food and alcohol consumption resulting from

access programmes in Nepal, negative outcomes from increased mobile phone use in

Bangladesh.

Identified emerging social issues e.g. chronic indebtedness in Indonesia, increasing alcohol

abuse in Nepal, abandonment of elderly in Bangladesh and Nepal, missing middle generations

in Mozambique, Ghana, Nepal and Indonesia, the need for re-evaluation of gender and ethnic

based quotas in Nepal

Explored the disconnect between reported data and reality e.g. exaggerated numbers ofattended births in Bangladesh, over-reported school attendance in Indonesia and Bangladesh,

under-reported teacher absenteeism in Nepal, Bangladesh and Indonesia

Provided more nuanced understanding of statistical data e.g. increasing drop out of boys from

school in Bangladesh and Indonesia due to constellation of factors including lack of incentives,

privileging girls, punishment skewed towards boys, too few male teachers, pull factor of

unskilled jobs and the desire to earn for their own recreation and status (e.g. phones, clothes,

motorbikes)


8/22

7

Helped design culturally appropriate household survey questionnaires e.g. in Nepal and Ghana

which recognise local terms and customs, provide wider and more appropriate choices of coded

answers

Provided the basis for developing peoples Theories of Change to compare with theoretical

programme Theories of Change e.g. Nepal Koshi Hills and Nepal Rural Access Programme.

Put development interventions in context. Because RCAs take a cross-sectoral stance, some

interventions are seen as unimportant or too small to have any significance for people. e.g.

chicken immunization in Mozambique was not a priority for people who very rarely ate chicken,

stipends for primary education in Bangladesh were burdensome to access and too small to

make a difference.

Provided frontline service providers with chances to share their frustrations e.g dispensary staff

in government health facilities in Bangladesh and Nepal, over-burdened midwife services in

Malawi, teachers time spent fulfilling many non-education related tasks in Bangladesh,

Indonesia and Mozambique

Explored difficult or sensitive subjects e,g, ethnicity, domestic violence, political empowerment,

crime, sexuality and interacted with small or marginalised groups which may be missed

through randomised surveys.

Allowed people in positions of authority to freely discuss sensitive topics, e.g. in Indonesia

people in authority tend to say the reason for children not going to school is lack of money while

when approached more informally they opened up about other factors such as pre-marital

pregnancy and truancy.

Ultimately, RCA studies are only useful if they are taken seriously by policy makers. The revelation of

unexpected and surprising realities often has a powerful effect on policy makers. Providing un-filteredvoices and experiences of people living in poverty helps policy makers put real faces to the anonymous

numbers often provided by other forms of research. While the dominant rubric of analysis-think-

changepervades, there is value in complementing this with findings which promote see-feel-change

(Kotter and Cohen, 2002). Evidence suggests that when complementary immersion programmes are

offered in parallel to RCA studies for policy makers, the understanding and response to the RCA study

increases.

1.5. LIMITATIONS

RCA studies are generally small scale. They deliberately set out to explore the range of experiences

of people living in poverty, so are not intended to be generalizable. RCA studies provide depth rather

than breadth. As such they cannot replace other research, monitoring and evaluation methods but caneffectively complement them.

As the RCA purposely seeks out multiple realities and listen to peoples own voice, it does not lend itself

to providing policy recommendations. Rather, RCA researchers prefer to present policy makers with

impl icat ions collated from the RCA study findings. By doing this, they avoid the problem of outsider

interpretation, infiltration of authorial voice and confirmation bias. However, policy makers often feel

challenged by being asked to reflect on study implications rather than a set of normative

recommendations.


9/22

8

2. CONTEMPORARY RELEVANCE

As the 2015 MDG deadline approaches, donors and recipient Governments are increasingly expecting

the development community to provide rigorous and scalable approaches to evaluation to prove

effectiveness of interventions. Much emphasis has been given to evidence-based programming with a

concomitant swing towards commissioning experimental and quasi experimental evaluation design.While helpful in determining what works, these approaches rarely address issues of whyand what else

might be just as ef fect ive (or more effective). Experimental/quasi experimental evaluation are

necessarily large scale (to meet statistical significance criteria), expensive and generally slow to yield

results. By contrast, RCAs can provide alternative, complementary, relatively quick and low cost

research-based evaluation directly with the people most affected by development interventions. The

RCA can provide windows on the aspirations of ordinary people and can provide space for comment

on experienced change.

2.1. AID EFFECTIVENESS

Since the 2005 Paris Declaration on Aid Effectiveness, attention

has been given by donors to improving the coordination,

effectiveness as well as developing country ownership of

international aid and poverty reduction efforts. One unintended

effect has been that emphasis has moved upstream with a

concern for efficiencies and better aid management, The so-called

more with lessagenda pervading aid delivery has led to widening

gaps between people on the ground and decision makers. The

relevance and responsiveness of programmes is consequently at

threat. The RCA provides a way to re-connect the worlds of policy

makers and programme management with the realities of people

living in poverty. Understanding how programmes actually operate

and are experienced on the ground is essential to developingoperational know how, improving programmes and maintaining

creativity and innovation.

In Provocations for Development(2012), Robert Chambers notes

'agreement seems universal that immersions give insights and

experiences that are not otherwise accessible...the world can be

seen from the other way round, from the perspective of people living

in poverty'. Immersions serve to re-connect development

professionals with people. But these are rarely undertaken .It

seems that finding the time, commitment and courage to spend time

doing an immersion is limited. Where policy makers and

development professionals are unable to undertake immersions

themselves, RCAs can provide an alternative re-connection by

proxy.

If commissioners of RCA research

can themselves give time to even

one immersion, their connection to

the reality of people living in

poverty will be immensely

enhanced.

Members of the health andeducation Sector Wide Approach

consortia in Bangladesh were

provided with opportunities for

facilitated two night immersions

with families living in poverty in

year 3 of the RCA longitudinal

study. This led immersees to

check some of the findings for

themselves. The resonance of

their own, albeit brief, experience

led them to trust the findings and

promote the findings among their

colleagues. In effect they became

champions for the RCA.

IMMERSIONS ON THE

SIDE


10/22

9

2.2. VALUE FOR MONEY

Alongside the push for increased aid

effectiveness there has been an increased

emphasis from many donors on Value for

Money (VfM); reflecting tax payers/voterconcerns in donor countries on the scale and use

of development aid budgets. RCA can contribute

to VfM in two key ways; RCAs are cost efficient

means to find out and findings can lead to

design of more effective and better targeted

programmes.

The original Bangladesh RCA was relatively

costly because it was pioneering the approach.

It involved development of a careful iterative

approach, combined with the testing and

promotional exercises necessary for a newprogramme. More recent RCA budgets have

been much lower, with a single RCA study1 ,

including training of researchers and piloting the

approach for the country context as well as

leadership and quality assurance provided by an

international team leader, costing in the region of

US$35,000. This figure compares favourably to

Annual Reviews, conducted for donors such as

DFID which generally cost more than twice this.

In terms of value of outcomes, the RCAunequivocally provides insights into what people

living in poverty need and value. Understanding

what change people hope for means better

understanding, leveraging, tailoring and

prioritising complex and high cost programmes

and investments, leading to better VfM within

these programmes.

Cardamom growing in Nepal

1A discrete and complete one time study rather than a longitudinal study

An example of how the RCA can contribute to better

VfM in programmes comes from the Nepal Koshi

Hills RCA. The RCA was a single discrete one time

study looking back at 40 years of development

assistance. It revealed that peoplesperceptions of

what was significant as well as their expressions of

aspirational change were not necessarily in synch

with the priorities of development donors.

Furthermore, the RCA revealed that the heavy

sectoral focus of most development programmes

kept them from identifying significant community-

level changes.

For example, the RCA demonstrated that production

of cardomom (a culinery and medicinal spice) had

become very lucrative. Villages designated as

extremely poor were in fact found to have thriving

cardamom industries and families with significant

disposable incomes. Neither the national statistical

data nor development programmes had up to date

information on this transformation. Yet its

introduction and spread, more than 10 years ago,was driven by people in communities themselves.

Development agencies were slow to pick this up and

have only relatively recently offered the needed

support.

Similarly the rapid increase in overseas migration by

people from the area was spearheaded by people

themselves by making and exploiting their own

networks. The official development assistance for

this has lagged behind.

In both cases the lag has been between 7-10 years.

Reducing this lag and making these local initiatives

work better represents a major VfM saving.

RCA CONTRIBUTION TO VFM IN NEPAL


11/22

10

2.3. OWNERSHIP:WHOSE VOICES COUNT?

There is a growing concern among development aid stakeholders that ownership, identified as a key

element for aid effectiveness by the 2005 Paris Declaration on Aid Effectiveness, is too often confused

with recipient Government ownership. Since the Accra and Busan conferences, civil society

organisations have been increasingly included in evaluations and research, but those in villages, townsand cities who are the intended target group of the development interventions are often still forgotten.

The assumption that recipient governments and civil society institutions can fully represent the views

and needs of local communities is recognised by many to be flawed2.

Studies indicate that people, particularly those living in poverty, rarely feel able to make their voices

heard. There are few opportunities for people to voice demands and concerns and little time/inclination,

access or capability to participate in conventional formal mechanisms and platforms where voices may

be raised. Many ordinary people eschew public forums for raising issues fearing that they may not be

heard, may be ridiculed or singled out. These public spaces are often dominated by political interests

and the prevailing power dynamics may preclude open discussion on issues. Often people simply do

not have the time to engage in this kind of policy dialogue platforms or have little optimism about the

outcomes. Even though many development interventions have been aimed at creating additional'invited spaces'for civil society participation these may be manipulated by those inviting (often national

and local governments) and/or co-opted by powerful interest groups. The lack of transparency around

who is invited and who is not as well as the politicisation of public space furthers fuels frustration and

reluctance to use these spaces by ordinary citizens.

The RCA can provide an alternative which can amplify ordinary voices. It provides an easy trusted

space for emergence of peoples real perspectives and opinions rather than ones shaped by desire to

please, conceal or manipulate for gain which often manifest themselves in public shared spaces. It

enables detailed triangulation to gauge the veracity of what people say. For example, the Ghana RCA

revealed parents were very upset about teacher absenteeism but feared losing the few teachers they

had if they complained. Patients in Bangladesh laughed at the notices in hospitals suggesting that theylodge complaints with the hospital administration, Why would we do this and put at risk future

treatment?.

The RCA provides opportunities for local service providers to also share perspectives while chatting

informally. Teachers in both Bangladesh and Indonesia shared many insights into the manipulation of

public exams during the RCA but would not be prepared to share this in public forums or in meetings

convened by their respective education department. These forums, they explained, were not times for

discussion or listening to our views but means to give us instructions from higher authoritiesMedical

staff in Nepal and Bangladesh shared their frustration with their governments free medicines schemes

which put demands on them to dispense medicines to the non-ill, wasting time and resources but ,

again, had not been able to raise this in conventional forums.

To redress this situation, development partners such as DFID, Sida and DFAT have indicated in recent

years that they need to better understand how aid interventions are viewed and experienced by local

populations, to get closer to understanding what works and what does not, and why. RCAs included

routinely as a complement to information gathered through their representative organisations (elected

representatives, membership organisations etc) provides a means to cross check and expand the scope

of opinion gathering.

2Easterly, W.R. The White Man's Burden: Why the West's efforts to aid the rest have done so much ill

and so little good. New York: Penguin Press, 2006.


12/22

11

3. RCA STUDY DESIGN &IMPLEMENTATION

A major challenge for RCAs come inherently with their goal to shed light on and understand multiplerealities. RCA purposively engages with the complexity; diversity of contexts, opinions, experiencesand suggestions. This poses a huge challenge to deliver consistent, comparable and coherentinformation to a client. Experience of conducting credible and well accepted RCAs has indicated thatthere is a need for careful design to deliver the right amount of flexibility and reflexivity to minimize biasand maintain objectivity.

3.1. CONVERSATIONS

Much of the strength of the RCA lies in its flexibility and

responsiveness to context and moments. Conversations are a

relaxed and informal way to interact with people and to use

successfully require an emphasis on the role of attitudes and

behaviour which permeate all human interactions. These

behaviours include unequal power, biases, assumptions, pride,

status, commitment and interest which shape the rules ofengagement and can lead to mis-understanding, distortion,

manipulation and concealment of information on both sides. This

emphasis distinguishes the RCA from other methods of

engagement where power dynamics are difficult to diminish.

Having time to build trust and familiarity has proven again and again

to help people to relax and open up. Staying in peoples own

houses and fitting in with their routine, their rules and becoming

learners contributes greatly to reducing power gaps which

normally exist between people and outsiders. Information arising

from informal conversations is rarely directly recorded either in

written or in audio-visual form. This further enhances the quality ofthe interaction and the trust and confidence in the engagement

process.

Conversations can be curtailed when people lose interest, can be picked up again at a later time , can

be shared and referred to with others at other times and can take place while people are going about

their usual routine. Time and informality confer RCA considerable comparative advantage over many

other research methods.

Conversations are sometimes augmented with shared

drawings, diagrams, photographs or dramatized explanations.

These are not the core of RCA but can sometimes help the flowof conversation. At times conversations take on a life of their

own and the researcher may become an observer of debate

and animated discussion.

There is nevertheless a need to ensure that most

conversations have a purpose.

Drawings of ty pes of teachers that

children did not like which sparked

animated conversation about changes

result ing from the teacher training

programm e- Bangladesh 2009

One woman in the

household was very

uncomfortable with me.

She barely greeted me

when I arrived and kept

away. But next morning

we worked together in the

field and she became

relaxed. Over the rest of

the time I spent with them,

she gradually opened up

more and more.

Nepali RCA researcher (male),2014

IT TAKES TIME


13/22

12

3.2. AREAS OF ENQUIRY

RCAs purpose is achieved through detailed briefing sessions with the RCA team members before going

to the study locations. Usually based on input from commissioners of the research and extensive

literature reviews, the team leader facilitates a review of the thematic basis of the RCA, in particular

identifying contradictory evidence, gaps in evidence and understanding as well as ambiguous findings.Areas of enquiry are developed sometimes through the generation of a peoples perspective theory of

change diagrams. Care is taken not to fuel assumptions or pass on received wisdom as the researchers

operate best when they can have open conversations. Based on these areas of enquiry, loose

checklists are developed by research team members to provide an aide memoire for their

conversations and to help team members probe and clarify issues. Such checklists are not research

questions and are not prescriptive as this inhibits the natural flow of conversations.

Example Checklist themes from Previous RCAs

Indonesia (with focus on the Basic Education)

Value of education: how parents and children perceive and value education?

Quality of education: how parents and children perceive quality?

Access to education: what are the issues behind non-attendance and drop out?

Enabling and hindering factors for education

Parental involvement: in schools and with their children's education generally

Bangladesh (with focus on the Health and Education services)

Rights: How do people understand their rights and try to operationalise them in relation to

services? Knowledge: How well do people understand how public systems work, their entitlements and

who is responsible for what? How do people deal with the informal aspects of the systems?

Choice; what influences choice of service provider ( formal and informal)

3.3. THE TEAM

Experience from previous RCAs indicates that the RCA team should comprise of people with high levels

of enthusiasm, appropriate and sensitive attitudes towards people living in poverty and willingness to

suspend judgment and challenge their own biases. They must also be willing to subject themselves toquite difficult living conditions in the field (sleeping on the floor or sharing with host household family

members, basic bathing and toilet arrangements (if any!), rats, extreme heat/cold, insects, etc).

Students and recent graduates (with anthropology, sociology, development studies, journalism

backgrounds) often make the best team members.


14/22

13

TRAINING &PILOTING

Since the skills, attitudes and attributes required differ from other forms of qualitative and action

research, it is essential that the team engages in a short orientation and capacity building process (4-

5 days) which includes pilot testing through a minimum of two day and two night immersion experience

(mini RCA). Even with an experienced RCA team, piloting should be considered when moving into a

new location or new areas of enquiry.

The pilot is intended to have the following objectives:

to further develop the RCA for each new context and the needs of the programme

to build capacity of the RCA team to acquire the attitudes, behaviour and skills to be part of the

main RCA Study and begin (or continue) the process of building sustainable capacity to conduct

such studies in the future.

team building

to promote the approach among stakeholders, building their interest and support for the

initiative

3.4. MAINRCASTUDY

The RCA study comprises the following steps

STEP1:BRIEFING

This always involves joint discussion and development of the areas of enquiry (see above),careful

decisions around which locations and households to select for the study and a review of the teams

own biases and assumptions. This latter is intended to make these explicit and take action to

minimize the influence of these in the conduct of the study.

The selection of locations for the RCA is challenging but RCA teams need to resist the pressure which

might exist to 'sample'. As the RCA intentionally seeks to understand multiple realities and does not

claim to provide representative findings, conventional sampling is inappropriate. Nevertheless in most

RCAs conducted to date, criteria are developed for some kind of purposive selection of study locations.

In mixed methods approaches where RCTs have been used, the RCA has also chosen to study

Detailed briefing, selection of

locationsimmersion

detailed de-briefings

immediatelyafter immersion

analysis&reflexiveprocessing

report &communicating

findings


15/22

14

treatment and control areas but has taken additional criteria into consideration too. There can also be

an argument for 'no criteria'so that reality is not distorted. The aim to reveal multiple realities can be

used to determine the location selection criteria which therefore embrace diversity (livelihood, ethnicity,

remoteness/access etc).

Flow diagram to show process of selecting locations

STEP2:IMMERSION

Immersion is core to the RCA and changes the dynamic between the researcher and the host families.

It involves not just staying overnight but also taking part in daily life. This creates opportunities to

experience first-handdetails of peoples lives(e.g. about the diff iculties collecting water, the behaviour

of service providers, the poor light with which to study, the impact of continuous rain on their livelihoods,

the hardships tilling drought affected land etc). It also provides the opportunity for detailed observation

(e.g. of household dynamics, child-parent behaviour and relationships, daily routine, who does what,

the difference between what people say they do and actually do). But most of all, it enables a relaxed

and trusted context for conversationsand enhanced understanding of how people live their lives By

staying with a family, the power distance between the researcher and the family diminishes. By

purposely staying with the most disadvantaged, other voices (older people, children, poorer,

marginalised) often 'unheard' in conventional evaluations are also privileged.

Host households are a key focus for the RCA and host the immersion. Often a household is defined as

'a family unit which cohabits around a shared courtyard and often cooks together'but there are countryand context specific variations to this. The selection of host households in most RCAs to date has been

purposive, often focusing on poorer households or households with particularly relevant socio-economic

or demographic characteristics. For example, in Indonesia study households had to include school-

age children, whereas in the Nepal Koshi Hills study, households where several generations lived

together were important for the task of reflecting on forty years of development interventions.

Often an advance team facilitates identification and

explanations of the purpose of the RCA to prospective

host households. Some experienced RCA teams

have entered communities cold i.e. without any

advance planning and this can work well too if

handled sensitively. RCA team members are alert to

gatekeepers and balancing the tensions between

paying courtesy to power holders but staying true to

the objectives of interacting with ordinary people in

ordinary ways. They work hard to reduce

expectations, explain their purpose and the need to

engage with reality, which, of course, also requires

them not to be afforded guest status.

Our experience with RCA has shown that as repeat

visits are made each year, it becomes easier and easier for the researcher to fit in with family routine.

The opportunity to share sleeping quarters (usually in our experience sharing a bed with family

Researcher shares a bed with family in

Bangladesh

external criteria e.g.areas whereprogramme

does/does notoperate

negotiated variablese.g distance from

urban centres,ethinicity,'control' &

'treatment'

RCA team makesown reconnaisance

and selection oflocations


16/22

15

members or sleeping on mats on the floor) confers some degree of equity in the relationship and

demonstrates a real desire on the part of the researcher to experience normal life in the household. No

RCA has been undertaken with less than two nights spent with the family per visit and this should be

considered a min imum. The first night often retains a novelty feel and the second is more productive

in terms of opportunities for open discussion.

Host households participation must be voluntary and people unfamiliar with RCA are often sceptical of

the willingness of families to host researcher and the ethics of burdening poor people with another

'mouth to feed'and a distraction from daily survival activities RCA experience to date has shown that

host families welcome the researcher and enjoy opportunities to chat in their own homes. Comfortable

in their own space and time, household members inevitably open up and readily explain their views

and their feelings, provide examples of change, debate and discuss issues of concern in ways which

they would rarely do in public forums or formal surveys. They appreciate showing the researcher what

their life is really like and the interest shown in them.

The issue of compensation to host households has been one over which all the RCA teams have

deliberated long and hard. The principle adopted is to ensure the family does not incur any cost from

the researchers visit. It also recognises that any contribution must bemade discretely to ensure noloss of face and to minimise local jealousies. Past experience in other countries3suggests that the most

appropriate form of compensation are basic food or consumable items (e.g. rice, cooking oil, salt, soap)

given to the host household upon leaving. These may be supplemented or substituted with other small

items e.g. the coloured pencils, torches, emergency lights, mosquito nets or blankets which the team

has taken with them. Cash is neither appropriate nor expected and has the danger of commoditising

the relationship. Copies of the photos taken of the family are given to them on subsequent visits and

are regarded as important gifts.

RCAs always involve interaction with people beyond the immediate household, including neighbours,

others in the community and local level service providers. This provides further opportunities for

triangulation and quickly expands the scope of the study. It is not unusual for a small scale RCA study

to include in depth conversations with well over 1.500 people.

STEP3:DE-BRIEFING

Ask me what issues I was working on 8 years ago and Id struggle to remember the details,

but I remember every detail of my immersion experience 8 years ago and how I felt

throughout. Subrata De, Christian Aid

The most effective means of collating the combined experience of the RCA team members is through

very detailedde-briefings which the RCA team leader and sub-team leaders facilitate soon after the

completion of each RCA location study. This provides an opportunity to maximise the sharing and

recording of data as information is compared and added to by different team members during such de-briefings. Team members recall observations and experience which they may not have noted in a

written report. This joint activity is an important additional opportunity for triangulation of findings. The

quote which starts this section illustrates how the combination of conversations, observation and

experience leads to exceptional recall. As a Ghanaian RCA practitioner noted, I lived it so I

remember it.

3Confirmed during the evaluation of the RCA in Bangladesh.


17/22

16

STEP4:ANALYSIS AND REFLEXIVE PROCESSIING

The established approach of Framework Analysis can be used to examine the large quantity of

observational and conversational data collected using the following steps:

Identification of thematic framework: this phase involves identifying key issues, themes and

categories raised by the respondents which emerge from the discussion phase.

Charting: this phase involves re-visiting the entire set of data and placing summaries of the

views and experiences shared by the respondents inside the chart of themes. (categorisation)

Interpretation: this phase attempts to draw inferences from the charted summaries. However,

caution is always exercised in RCAs not to overlay the researchers interpretation of the

information. As the RCA is intended to explore multiple realities, it is important not to over-

categorise and to ensure 'deviant' experiences and observations are also captured.

Software programmes such as NVivo can be used to manage the huge volume of qualitative data.

Some RCA studies construct grounded theory to provide further rigour to the interpretation of findings.

This involves a systematic building of theory (in reverse) through grouping and categorising ideas,concerns and experiences of people into conceptual theoretical frames to explain action and behaviour.

A key element of the analytical phase is facilitating reflexivity. This is a purposeful and resource

intensive attempt to reduce bias and subjectivity in the research process and analysis of findings. The

team leader leads the whole team through processes which remind members to question the meanings

and constructs they have deduced from their findings, to think about their own positionality and inf luence

on the generation of information. Together, the team questions interpretation and by bringing their

multiple perspectives and findings to the forum allows for collaborative minimisation of bias.

STEP5:REPORT AND COMMUNICATING FINDINGS

Communicating the findings of RCA studies which always provide a wealth of information is a challenge.

This is another point at which research or research commissioning bias needs to be recognised and

contained. RCA reports are supposed to convey the opinions, perspectives and experiences of people.

Authorial voice and outsider interpretation needs to be minimised or transparently acknowledged. There

is always differences of opinion regarding the depth and breadth of information to be included in reports

and it is probably best to provide a range of products including full detailed report, summary and policy

briefs.

Report writing, like the analysis, is usually a collective activity, even if there is a main author. This

ensures that the whole team agrees on the main findings, agrees on the way these are conveyed and

has a say in the final report format. Illustrative examples are written by team members themselves

based on their first-hand experience. As mentioned above, RCA teams try to avoid being asked forrecommendations since these require a level of interpretation inconsistent with the intention to convey

peoples reality. Sometimes, RCA participants themselves have volunteered recommendations and

these can be on-conveyed as long as the context for these is clear.

The wealth of visual material produced during the course of the RCA study (photographs, drawings,

diagrams, video) provides opportunities for multi-media approaches to communication. Photographs

taken by people living in poverty themselves have proved to be powerful tools for influencing policy.

Inclusion of stories, quotes and reported conversations add to the richness of information. The

combination of narrative and visual has been shown to widen audience appeal and get RCA findings

noticed.


18/22

17

4. NOTESFORRCASTUDYCOMMISSIONERS

It is important to make the intended use of the RCA data explicit. Is the RCA a pulse taking exercise?

Should it shed more light on previous research, or help interpret quantitative studies? Or is it a stand-

alone study of how people experience change or is it part of a mixed methods evaluation or research?

Broad areas of enquiry can be provided in Terms of Reference for further elaboration by the team. TheRCA team will want to maintain the confidentiality of the locations and households and so, while criteria

for location selection can be negotiated between the client and the team, actual identification needs to

be carried out by the RCA team themselves and they need to be able

to operate independently of any programmes or organisations.

In responses to tenders, the client should expect that the RCA

includes immersion and makes clear that engagement with people is

informal. The client should expect that the RCA team adopts

positionality aligned with people living in poverty. Sufficient days

need to be factored into the study for detailed and respectful

interactions with people.

Qualitative and quantitative research do different jobs and need to be

judged accordingly. It needs to be remembered that research

judgment is involved in both. Qualitative researchers are often

expected to go the extra mile to prove rigour/soundness. The client

should expect RCA study designs to make provisions for assuring rigour. Rigour is a means to

demonstrate plausibility, credibility and integrity and ultimately confidencein the research. The client

should require RCA design to demonstrate

rigour in procedures (e.g. justifications for site and household selections, allocation of sufficient

resources for meticulous reflexivity, demonstration of good triangulation practice and inclusion

of contradictory information) rigour in analysis (e.g. efforts, where appropriate, to allow participants to review and validate

findings, transparent efforts to abstain from interpretation and judgement)

rigour in documentation: (e.g. efforts to keep complete records and provide a process trail of

the research (sufficient for another to follow and come to same conclusion), transparent and

explicit narratives).

Clients need to be aware that RCA produces rich and detailed information. To avoid over simplification,

report length should not be unrealistically limited. RCA reports will include visual and apt illustration.

The client should consider requesting a number of discrete products if they have concerns about the

readability of long reports rather than compromising on detail. As discussed above, clients should not

expect generalizable findings nor recommendations, but can expect RCA report authors to compile

implications from their findings.

5. FROMRESEARCHTOACTION

RCA findings often challenge assumptions that have been central tenets of local and national level

policy making and operational planning. Whilst for some this can be an alienating experience, for many

policy makers in Bangladesh and Nepal engagement with the findings of the RCA has been a catalyst

for significant shifts in thinking on certain policy issues. In most cases the true impact of this shift will

not be felt by ordinary people for some time to come. The following are a few examples of how RCA

study findings have been deliberated upon and used.

No research is free of biases,

assumptions and the

personality of the researcher

Sword, 1999

BI S


19/22

18

BANGLADESH

A series of workshops were conducted in February 2012 at the conclusion of the Sida supported

Bangladesh RCA longitudinal study with a range of local service providers (teachers, health providers)

who had been involved in the five year RCA study (2007-2011). They shared experiences and

endorsed the findings and felt empowered (by the research and the realisation of common concerns)

to raise some of the issues within their institutions.

The Department of Education and other education stakeholders have taken seriously the perspectives

provided by through the RCA around the primary school stipends programme and peoples preference

for school feeding programmes. Perhaps most significantly there has been recognition by

representatives of the Government of Bangladesh and SWAP partners that the design of the PEDP-III

(Bangladesh National Government Primary Education Development Programme) was directly informed

by the findings of the RCA reports; particularly in relation to the complexities surrounding school boy

dropout rates. New measures to encourage and retain boys in school are explicit in the new programme.

Sida, as a relatively small bilateral donor in Bangladesh, used its comparative advantage within the

SWAP donor consortia, derived from its interest in accountability, voice and rights, to introduce and

promote the RCA. The RCA was envisioned to improve programme effectiveness through providingnew types of information from peoples perspectives at the grassroots. As one former Sida Embassy

staff member remarked in the reflection interview: the Reality Check made us more visible, and it gave

us something to bring to the table

Both the European Union and Sida referenced and used the RCA findings to develop their respective

Action Plans on gender and gender-based violence in Bangladesh, whilst the World Bank reported it

provided RCA reports to all Education and Health consultants working in the country as required

reading.

NEPAL

In Nepal, the DFID supported Koshi Hills RCA findings were shared with the National PlanningCommission (NPC) in a series of workshops in 2013. The reception was extremely positive and has led

to a demonstrable demand within the NPC for qualitative research studies to complement the dominant

tradition of quantitative surveys. A new knowledge sector programme has been designed to, among

other things, meet this demand. DFID has commissioned a further longitudinal RCA to assess change

in the major rural access programme in the mid and far west. This has influenced the design of the

quantitative household survey design and has flagged up issues for special studies e.g. the changing

face of poverty.

6. THE RCACOMMUNITY (TO DATE)

Whilst the RCA was an initial endeavour between the Swedish Government and GRM International inBangladesh, the positive response of donors has led to the RCA being replicated by a number of other

organisations. This loose group of organisations is now regularly collaborating as The RCA Community

of Practice, which includes a number of consultancies, research institutions and individual researchers:


20/22

19

The RCA website (http:/ /reali ty-check-approach.com/)has been set up as a forum for this

Community of Practice as well as for more general public access.

7. FUTURE USE

At the time of publication, RCAs have been used as

Stand-alone longitudinal qualitative studies e.g. Bangladesh, Mozambique

One-off retrospective study e.g. Koshi Hills, Nepal

Pulse-taking study e.g. Indonesia

Scoping study to inform the design and implementation of larger scale household-based, mixed

method study e.g. Mid and Far West Nepal

Integrated within mixed method third party longitudinal evaluations e.g. Ghana and Mid and Far

West Nepal

It is hoped that the use of RCA will continue to grow as new development partners appreciate the value

of the approach. The application of RCA has already gone beyond its initial utilisation as a standalone

longitudinal study (as noted above). Involving the RCA in major mixed method longitudinal studies is a

new innovation and allows for an optimal method mix. Initial longitudinal studies of this nature were

funded by DFID and have built mutual respect between traditionally separate disciplines of qualitative

and quantitative research.

The increased interest in beneficiary feedback mechanisms (BFM) to improve development

programmes provides another opportunity for RCA. RCAs informal and non -intrusive approach lends

itself to integration in methods to understand beneficiary perspectives and allow people to be critical

and demand their voices are responded to.

Additionally, decentralisation agendas in many countries offer opportunities for RCAs. Local

governments with restricted budgets will be able to afford small-scale but in depth RCA studies whichhelp to inform them to make their programmes context specific and responsive. This was an important

consideration in the design of the Indonesia RCA+ project, where local level research organisations will

be trained and mentored to provide this kind of research service.

While much is made of the way RCA can amplify the voices of people living in poverty, there is more

that can be done to strengthen the voices of frontline service providers, who are also marginalised from

policy dialogue space. Staying in the homes of local teachers, midwives, police, local government

officials can potentially provide important insights into their experienced reality and the constraints and

frustrations they face.

RCAs to date have been commissioned by development partners around education, rural access,

health, agriculture, social protection and rights based programmes. However, there is scope for privatesector and corporate social responsibility programmes operating in complex environments (e.g. in the

extractives industry) to commission RCAs to connect programmes to ground realities and to better meet

peoples aspirations, mitigate environmental and other risks.

References:

Chambers, R 2012, Provocations for Development, Practical Action PublishingKotter, J.P.and D.S.Cohen, 2002 The Heart of Change: real life stories of how people change their organisations

Harvard Business Review Press
http://reality-check-approach.com/http://reality-check-approach.com/http://reality-check-approach.com/http://reality-check-approach.com/


21/22

20


22/22

21

The 'A Short Introduction to RCA' is a Effective DevelopmentGroup production finalised in collaboration withthe Indonesia Reality Check Approach + Project anAustralian aid project, funded by the Australian Department ofF i Aff i d T d

rca introductory note 2014

Documents