IHSN Question Bank- supporting quality and consistency

in survey data

Olivier Dupriez World BankFrançois Fonteneau PARIS21/OECDGeoffrey Greenwell PARIS21/OECDMark McConaghy DFID - UK

International Household Survey Network

Evidence informed policy

“Trusted, high quality statistics are a necessary condition for developing and maintaining a strong

economy and society that functions well”

(Hon. Prof. Peter Anyang’ Nyong’o former Minister for Planning and Development, Kenya)

“Comparing groups, cultures, nations or continents is an essential means of

distinguishing between local conditions and universal regularities”

(Janet Harkness; Cross-Cultural Survey Methods; 2003:3)

Concepts

Latent constructs

Indicators

Questions

Questions and concepts

Improved/ unimproved drinking

water sources

Access to clean/safe

water

What is the main source of drinking water for your household?

Disease prevention

….but different building blocks

Joint Monitoring program

What is the main source of drinking water for members of your household?

 

11 Piped – to dwelling

12 Piped – to yard or plot

13 Piped – public tap/stand pipe

21 Tubewell/borehole

31 Dug well – protected

32 Dug well – unprotected

41 Water from spring – protected

42 Water from spring – unprotected

51 Rainwater collection

61 Tanker truck

71 Cart with small tank/drum

81 Surface water

91 Bottled water

96 Other (specify)

Typical surveyIn the rainy season what is the main source of

water for drinking and cooking for your household?

 

1 Private connection to pipeline

2 Private well

3 Public tap/standpipe

4 Public well

5 Neighbors

6 Water vendor

7 Spring

8 River, stream, lake, pond

9 Rainwater

10 Other (specify)

11 Bottled

   

   

   

Changes to questions

Joint Monitoring Program

What is the main source of drinking water for members of your household?

 

11 Piped – to dwelling

12 Piped – to yard or plot

13 Piped – public tap/stand pipe

21 Tubewell/borehole

31 Dug well – protected

32 Dug well – unprotected

41 Water from spring – protected

42 Water from spring – unprotected

51 Rainwater collection

61 Tanker truck

71 Cart with small tank/drum

81 Surface water

91 Bottled water

96 Other (specify)

Typical surveyIn the rainy season what is the main source of water for

drinking and cooking for your household?

 

1 Private connection to pipeline

2 Private well

3 Public tap/standpipe

4 Public well

5 Neighbors

6 Water vendor

7 Spring

8 River, stream, lake, pond

9 Rainwater

10 Other (specify)

11 Bottled

   

   

   

Deprivation estimates in same country

74

63

17 15

6 5

14

70

46

15

26

613

17

0

10

20

30

40

50

60

70

80

90

100

Shelte

r

Water

Sanita

tion

Info

rmat

ion

Educa

tion

Health

Food

per

cen

tag

e

DHS 1998 MICS 2000

Different ways of treating household and communal information

Different illnesses included- untreated diarrhoeal

illness verses ARI

?

All aged 5 or over

Has (name) ever attended school

<If YES>

What was the highest school level attended?

Within that school level, what was the highest grade

completed?

Can you read and write?

<If YES and age greater than 13>

What was the highest school level you attended?

Within that level, what was the highest grade completed?

Measuring educational attainment

Educational attainment outputs

37.6

26.5 27

9

51.8

14.8

32.3

1.10

10

20

30

40

50

60

70

80

90

100

No schooling Primary Secondary Above secondary

pe

rce

nt

DHS 2004 HIES 2000

Biased up and down by those who cannot read & write but did

attend some schooling?

Documentation for replication

“The only way to understand and evaluate an empirical analysis fully is to know the exact process by which the data were collected and the analysis produced… The replication standard holds that sufficient information exists with which to understand, evaluate, and build upon a prior work if a third party could replicate the results without any additional information from the author”

(Gary King in Political Science and Politics Vol. XXVIII No 3 443-449)

Literal question –What kind of toilet facility do members of your household usually use?

Answer categories Flush/pour flushFlush to piped sewer system 11Flush to sceptic tank 12….

Ventilated Improved Pit latrine 21Pit latrine with slab 22Pit latrine without slab/open pit 23….

Post question text If ‘flush’ or ‘pour flush’ probe – Where does it flush to? IF NECESSARY ASK PERMISSION TO OBSERVE THE FACILITY

Example DDI elements

Notes for Interviewer Pit latrines are the best conventional sanitation option when there is no source of water on or close to the plot so that water use is low (typically less than 25 liters per person per day). They may also be used when water consumption is higher, provided that separate provision is made for sullage (“gray” water) disposal. However, they may be less attractive to users of pour-flush toilets in such situations. Simple unimproved pit latrines may smell. VIPs should be better but users may not appreciate the need for a dark interior and may modify the superstructure to provide more light, thus undermining the basic rationale behind the design. The VIP design is very dependent on a durable fly screen and may give rise to insect problems if the screen fails for any reason. High water table and/or flooding are obvious problems for pit latrines.

Universe All households

Source of information Responsible household member

DDI helps with interoperability

Inclusion of technical material

Ongoing and planned work, so far…

UNICEF/WHO immunization, anthropometry, nutrition etc

Joint Monitoring Program water & sanitation

FAO/ World Food Program Agricultural census and surveys

ILO economic activity, Int. labour migration

World Bank/ ILO consumption, savings & assets, remittances

World Bank/ ONS social capital

UIS/UNESCO educational surveys

IHFAN health facility surveys

• Using DDI to develop a meta data structure for data collection a priori, link concepts, indicators & questions

• Central repository for survey guidelines - custodians for topic areas (labour, health, food, education …) ‘a living resource’

• Generation of interviewing, editing and coding instructions

• Standards for coding and classification• Transparency with derived variables/ indicators creation• Tabulation plans for a primary analysis

Question Bank – key components

Question Bank – possible extensions

• Question Bank to include discussion forum• Facilitate production of questionnaires and data

input programs including CAI?• Provide the application to NSOs to help

establish Question Banks at country level• Introduce different language versions,

translation issues and cognitive testing/piloting• Develop information about relationship between

concepts and questions across countries and cultures

Technical aspects of the Question Bank

• XML schema has been developed to store all information (built on DDI and Dublin Core standards, plus additional elements)

• Q-Bank will be a user-friendly application that makes re-use of metadata easy (built on Flex/Air). It will be web-enabled and also work as a stand-alone application.

• user-friendly "metadata editor" developed to ensure easy maintenance of the system

• all applications are open source, and will be distributed free of charge

• XML schema and the application support multi-lingual metadata

• application can serve as a repository for international or national classifications

“Careful documentation of design decisions and implementations will enable cross-cultural researchers to begin accumulating a knowledge base of experience that will promote greater awareness of the relative advantages of each approach and move the community closer to establishing ‘best practice’ recommendations with confidence”

(Janet Harkness; Cross-Cultural Survey Methods; 2003: 34)

Question Bank important because

Thank you

www.surveynetwork.org

International Household Survey Network