Lessons from WEAI fieldwork in Uganda and Bangladesh

Presented by Katie Sproule & Chiara Kovarik (IFPRI)

A4NH Gender Methods Workshop, Rome

December 2-4, 2014

Why use cognitive testing and vignettes in the WEAI?

Introduction to cognitive testing

Introduction to vignettes

Applying these tools to the WEAI

The fieldwork

The results

Lessons learned

2

Why did the WEAI need these tools?

After the 2012-2013 baselines, it became obvious that the WEAI needed to undergo some revisions and streamlining

Key indicators were identified as problematic

Decision was made to develop a second version of the WEAI (WEAI 2.0)

Cognitive testing was conducted to ensure that the questions were capturing the various dimensions of empowerment and also to ensure that the index remained standardized

Vignettes were included to see if they would be a better way of getting at issues of autonomy

• The WEAI was developed by IFPPRI, OPHI, and USAID in 2012 to measure women’s levels of empowerment and inclusion in the agricultural sector

• It was initially designed to be a monitoring and evaluation tool for USAID’s Feed the Future (FTF) programming in the 19 FTF countries

• It is composed of 2 sub-indexes: the five domains of empowerment (5DE) and the Gender Parity Index (GPI).

• The 5 domains are: Production, Resources, Income, Leadership, and Time

3

• Cognitive testing is a qualitative

method that is paired with a

(quantitative) survey

• The purpose of cognitive testing

is to systematically identify and

analyze sources of response error

in surveys, and to use that

information to improve the

quality and accuracy of survey

instruments (Johnson, 2013)

• Cognitive testing can be

especially important for

new/revised instruments, or

those that will be used in

multiple country contexts

(Johnson, 2013)

• Generally conducted as a pre-test

before full field work begins

Cognitive Stages Cognitive Stage

Definition

Problems Causes

1. Comprehension

2. Retrieval

3. Judgment

4. Response

Source: Johnson, 2013

Cognitive process

Breakdown can occur in ANY of the four stages 4

Respondent

interprets the question

Respondent

does not understand

Unknown terms, ambiguous

concepts, long and overly complex

Respondent

searchers memory

for relevant information

Respondent does

not

remember/does not know

Recall difficulty, questions

assume respondent has information

Respondent

evaluates and/or estimates response

Respondent does

not want to tell, can’t tell

Biased or sensitive, estimation difficulty

Respondent

provides

information in the format requested

Respondent can’t

respond in the format requested

Incomplete response options, multiple responses necessary

How satisfied are you with your available time for leisure activities? Please give your

opinion on a scale of 1 to 10. 1 means you are not satisfied and 10 means you are very satisfied. If you

are neither satisfied or dissatisfied this would be in the middle or 5 on the scale.

Breakdown in comprehension:

Respondent may not understand the concept of “leisure”, or may understand it differently from the researcher

The concept of “satisfaction” is ambiguous and subjective

Breakdown in response:

Respondent may have never answered a question in this format. While questions with ranking scales are familiar to Western audiences, they may not be to everyone

Response error!!

5

Back to our example question: How satisfied are you with your leisure time? Please rank on a scale of 1-10, with 1 being completely unsatisfied and 10 being completely satisfied

Some follow up cognitive testing questions might be:

1. Can you tell me in your own words what “leisure” means?

2. What does it mean to you to be “satisfied”?

3. What recall period did you use in your response? Were you thinking about your leisure time in the past week? The past month?

4. Did you find this question difficult? If so, why?

5. Do you think others would find this question difficult? If so, why?

6

7

Conduct surveys with between 10-15 respondents per language group

Sampling should be done to maximize variance among respondents

At least two rounds of cognitive testing should be conducted

Enumerators need to be appropriately trained in cognitive interviewing

Audio-record the interviews

2 enumerators should be present for each individual interview

There is a large degree of flexibility in designing a cognitive testing that will depend on the survey and the context of the testing

8

What are vignettes?• Research method where respondents respond to a set of stories describing different

scenarios related to the topic for a hypothetical person/household

• The vignette provides enough context and information for participants to have an understanding of the scenario being depicted, but needs to be vague in ways that compel participants to ‘fill in’ detail

• Reveals perceptions and values, as well as social norms in the community

• Allows researchers to get at topics that might otherwise be challenging to ask about

• Can be used as an ice breaker, a way to close the interview, a stand-alone technique or part of a multi-method approach

9

ENUMERATOR: This set of questions is very important. I am going to

give you some reasons why you act as you do in the aspects of

household life I just mentioned. You might have several reasons for

doing what you do and there is no right or wrong answer. Please tell

me how true it would be to say:

[If household does not engage in that particular activity, enter 98 and

proceed to next activity.]

My actions in [ASPECT] are

partly because I will get in

trouble with someone if I

act differently.

[READ OPTIONS: Always

True, Somewhat True, Not

Very True, or Never True]

Regarding [ASPECT] I do

what I do so others don’t

think poorly of me.

[READ OPTIONS: Always

True, Somewhat True, Not

Very True, or Never True]

Regarding [ASPECT] I do

what I do because I

personally think it is the

right thing to do.

[READ OPTIONS: Always

True, Somewhat True, Not

Very True, or Never True]

G5.03 G5.04 G5.05

A Getting inputs for agricultural production

B The types of crops to grow for agricultural production

C Taking crops to the market (or not)

D Livestock raising

G5.03/G5.04/G5.05: Motivation for activity

Never true …………………………………..1

Not very true …………………………………..2

Somewhat true …………………………………..3

Always true …………………………………..4

Household does not engage in activity/Decision not made……………98

10

STORY QUESTION RESPONSE

A

The types of

crops to grow

for

agricultural

production

G4.A1 “[PERSON’S NAME] can’t grow other

types of crops here for agricultural

production. These are the only things

that grow here.”

To what extent does [PERSON’S

NAME]’s story describe your

situation?

Completely different………………….1

Not very similar…………………………..2

Quite similar……………………………….3

Describes my situation too …………4

Don’t know…………………………………97

G4.A2 “[PERSON’S NAME] is a farmer and

grows – [INSERT LOCAL CROPS]–

because her spouse, or another person

or group in her community tell her she

must raise these crops. She does what

they tell her to do.”

Whatever crops you grow for your

production, are you like [PERSON’S

NAME], doing what you are told by

others to do?

Completely different…………………..1

Not very similar…………………………..2

Quite similar………………………………3

Describes my situation too…………4

Don’t know…………………………..….97

“Now I am going to read you some stories about different farmers and their situations regarding different agricultural

activities. This question format is different from the rest so take your time in answering. For each I will then ask you how

much you are like or not like each of these people. We would like to know if you are completely different from them, similar

to them or somewhere in between. There are no right or wrong answers to these questions.”

11

1. Culturally and contextually appropriate

2. Focus

• Can make some parts more detailed or direct their attention to it (Braun & Clarke 2013)

• Vignettes should focus on “mundane occurrences” rather than disastrous events (Finch 1987, Hughes 1998)

3. Complexity

• Stay away from overly complex vignettes with too many characters (Braun & Clarke 2013)

• Ensure that the vignette is tapping a single one-dimensional concept (King 2014)

4. Ambiguity

Can intentionally make certain parts vague to explore assumptions (Braun & Clarke 2013)

5. Single vignette vs. staged vignettes

Presenting character or plot development in “stages” (Braun & Clarke 2013)

6. Number

Generally use 5-7 vignettes per concept to be measured (King 2014)12

Question:

1. Anchoring questions: “a technique designed to ameliorate problems that occur when different groups of respondents understand and use ordinal response categories” (King & Wand 2006)

2. Using “should” versus “would”: When asking about how a character might react you may want to get at moral aspects of the situation or the pragmatic (Braun & Clarke 2013)

Response:

1. Open-ended response vs. close-ended: Asking the respondent his/her thoughts on the vignette. Or giving response options (use an even number of categories)

2. Response categories relating to the hypothetical situation vs/ relating to respondent: Responses relate to how character in situation should or would act or relate to how respondent should or would act if he/she were in the same situation (WEAI 2.0)

13Enumerators need to be well-trained and comfortable with technique

Sites: Bangladesh & Uganda

Sample size: Consisted of 120 interviews in Uganda and 70 interviews in Bangladesh

Questionnaire: A series of roughly 100 questions were developed based off Johnson et al.’s (2013) paper on cognitively testing the original WEAI in Haiti

Photo credit: Chiara Kovarik15

Cognitive testing revealed issues with the following areas:

Distinction between different concepts

Time frame and recall issues

Abstract terms or concepts

Discrepancies between identifying something as challenging versus saying others would find it challenging

Photo credit: Katie Sproule 16

Original survey question: “Did you yourself participate in [ACTIVITY] in the past 12 months (that is, during the last [one/two] cropping seasons)?”

Cognitive question: “What timeframe did you include in your response?”

Problem: 35% of respondents in Uganda either could not come up with the recall period used or referred to a timeframe other than 12 months

Modified survey question: “Did you yourself participate in [ACTIVITY] in the past 12 months (that is, during the last [one/two] cropping seasons), from [PRESENT MONTH] last year to [PRESENT MONTH] this year?”

Results of modification: Timeframe recall errors dropped to just 6% in Uganda Photo credit: Katie Sproule

17

Pros from our experience:

Vignettes are fun and new!

Cons from our experience:

Some respondents found it challenging to understand the concept of a hypothetical situation

It was challenging for both enumerators and respondents to grasp what part of the story they were trying to relate to

Other thoughts:

Ambiguous results

Responses were often much longer and more descriptive than anticipated, even when posed in a close-ended manner (Bangladesh). This may have been due to not understanding the question.

Vignettes made an ideal candidate to cognitively test

What we might do differently next time

Vignettes as part of qualitative work?

18

Lessons learned and ideas for future research

• Vignettes and cognitive testing

are not for every questionnaire

• They take extra time, resources,

and enumerator training

• Cognitive testing was valuable in that it allowed us to understand

what is wrong with a question in a very specific way, rather than just

knowing the question is poor and should be changed; it answers the

how it should be changed

• Cognitive testing is not necessarily a stand alone technique; there

were areas where we are unsure what to make of the results (i.e.

effectiveness of vignettes vs traditional autonomy questions)

• While doing multiple iterations of testing may not always be feasible,

doing either a single iteration or a more extended pre-test could be

beneficial to survey designers (e.g. Haiti WEAI cognitive testing)

• It was especially important to cognitively test the WEAI, because it is

administered in 19 countries; similar testing should be considered

with other large multi-country surveys

19

Hopkins, D.J., King, G. (2010). Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability. Public Opinion Quarterly. pp. 1-22.

Johnson, K. (2014). “Cognitive Pretesting of Cross-nationally Comparable Survey Instruments in a Developing Country Context Seminar.” International Food Policy Research Institute. Washington, DC. 9 May 2014.

King, G. (2009). Anchoring Vignettes FAQs and Examples. http://gking.harvard.edu/vign/eg/ [Accessed November 6, 2014].

Wand, J. (2007). Credible Comparisons Using Interpersonally Incomparable Data: Ranking Self-Evaluations Relative to Anchoring Vignettes or Other Common Survey Questions. Available at http://wand.standford.edu.

Willis, G. B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage, Thousand Oaks, CA.

20

Thank you!

Any questions?

Contact Katie Sproule (k.Sproule@cgiar.org)

or Chiara Kovarik (c.kovarik@cgiar.org)

21

Bargaining power: “Hope is a cassava farmer in a nearby village. She has her own small plot that she works on, though her husband owns it. When it comes time to bring her cassava to market, her husband demands that she give him at least 80 percent of whatever she earns. What should Hope do?”

Mobility: “Sumi wants to visit her parents, who live in another village 20 kilometers away. Her husband agrees, but only if he goes with her. How much power does Sumi have to travel when and where she wants? Response categories: a lot; some; a little; none.”

Motivation for decisions-making: “Toko grows the crops for agricultural production that her family or community expect. She wants them to approve of her as a good farmer. Are you completely like, somewhat like, somewhat different or completely different from Toko?”

22

23

24

25