developing a high quality baseline salimah samji & mona sur world bank, new delhi june 21, 2006
TRANSCRIPT
![Page 1: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/1.jpg)
Developing a High Quality Baseline
Salimah Samji & Mona SurWorld Bank, New DelhiJune 21, 2006
![Page 2: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/2.jpg)
Overview
What is a baseline? Why you should care The phases of conducting a baseline
The errors to avoid in each phase How to manage the common errors Elements of a baseline survey (TOR)
![Page 3: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/3.jpg)
What is a baseline?
Fixing the time at the base – a benchmark from which you measure progress
Snapshot of indicators at a time Instrument used to:
Test hypotheses of project (assess results)Planning (refine targeting, indicators to
monitor)
![Page 4: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/4.jpg)
Why you should care …
To identify whether there were any benefits for the investments made Were objectives met? What factors explain the result? How can the program be improved?
Compare alternative models to get the biggest bang for your buck
To inform next generation projects Evidence-based policy making – demonstration
effect for government
![Page 5: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/5.jpg)
The phases of conducting a baseline
Design Implementation – actual survey Data entry and analysis Report writing
![Page 6: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/6.jpg)
Phase 1: Design Phase
![Page 7: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/7.jpg)
P1: Check-list
Clear objectives (what is the problem?) Clear idea of how you will achieve the
objectives (causal chain or hypotheses) Clear and measurable indicators
Clear (precise and unambiguous)Relevant (to objectives)Monitorable
![Page 8: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/8.jpg)
Example of a causal chain
Impact
Outcomes
Outputs
Activities
Inputs Facilitators, Revolving fund (credit)
Forming, federating and organizing SHGs
Number of SHGs, decreased input prices
Increased use of credit for income generation, Loan repayment rates
Higher income levels
Example: APDPIP
![Page 9: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/9.jpg)
P1: Check-list (cont’d)
Design survey instrument – keep it simple and related to the objectives and hypotheses you want to test
Link surveys to GIS – use consistent units Select controls/counterfactuals to attribute
change (causality) Timing of baseline
Before project (what if project never materializes)? 2 years into the project (intervention has begun)? Other factors (seasonality)
![Page 10: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/10.jpg)
P1: Check-list (cont’d)
Sampling strategy Random allows inferences about
a population Random Stratified random (include groups
which could be excluded)
Non-random – use a group smaller than the population
Introduces selection bias. Note: every stratification introduces a level of bias.
Population Size
Sample Size
10 10
50 44
100 80
500 217
1,000 278
3,000 341
50,000 381
1,00,000 385
![Page 11: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/11.jpg)
P1: Check-list (cont’d)
Over sample for attrition Design the database system for data entry Translate the questionnaire
Back translate to verify
Provide adequate training on The objectives and importance of the study How the sections are linked and what the questions
mean
Attend the training if possible (stay involved)
![Page 12: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/12.jpg)
P1: Check-list (cont’d)
Test the questionnaire Ask yourself, can you answer these
questions? Are they relevant to the outcomes? Will they be understood? Field test: To test both how the surveyor
administers the instrument AND how the respondent understands the question.
![Page 13: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/13.jpg)
P1: Why it is important to field test (i)
When asking a question about the level of awareness, the surveyor used a word that could mean awareness or knowledge – the respondent understood it to mean education. :
The question was: “Ram/ Gita knows about everything that happens (vikas) in the village. For instance, they know [the name of the sarpanch, when and where the Gram Panchayat meets, nature and
type of development work in village, etc.]”
![Page 14: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/14.jpg)
P1: Why it is important to field test (ii)
CASE STUDY: Domestic Violence Study in India
“In studying domestic violence, a question in the survey instrument asked if female respondents had ever been beaten by their husbands in the course of their marriage. Only 22 per cent of the women responded positively to this question – a domestic violence rate much lower than studies in Britain and the US had shown. In probing the issue with in-depth interviews we discovered that the women had interpreted the word ‘beating’ to mean extremely severe beating – when they had lost consciousness or were bleeding profusely and needed to be taken to the hospital. Hair pulling, ear twisting, etc, which were thought to be more everyday occurrences, did not qualify as beating. Reponses to a broader version of the abuse question, comparable to the questions asked in the US and UK surveys, elicited a 70 per cent positive response.”
Source: Vijayendra Rao (1998) – “Wife-Abuse, Its Causes and Its Impact on Intra-Household Resource Allocation in Rural Karnataka”
![Page 15: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/15.jpg)
Phase 2: Implementation
![Page 16: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/16.jpg)
What would you do if you …
Were a city person who didn’t speak the local language very well
Had to travel to several villages and spend hours asking people questions that have no relevance to you.
Were paid a small sum per questionnaire Not monitored by supervisors This is not your full time job
![Page 17: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/17.jpg)
The answer is simple
Sit at home or in a bar and fill out the questionnaires!!
![Page 18: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/18.jpg)
P2: Providing incentives and motivation
Sub-contracting surveyors from the state who speak the language
Include women surveyors Include a supervisor who conducts data
scrutiny If possible pay reasonable wages Randomly verify questionnaires to reduce
the likelihood of false responses (inform them beforehand - during the training)
![Page 19: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/19.jpg)
Phase 3: Data Entry and Analysis
![Page 20: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/20.jpg)
P3: Check-list
Data Entry Make the data entry system as fool proof as
possible - has unique identifiers to link both household, village and GIS data
Ensure database allows for merging of data Do not change/erase data on questionnaires Raw data should always be input as is, changes
can then be made in the database software (programatically) with documentation
![Page 21: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/21.jpg)
P3: Check-list (cont’d)
Often data entry is contracted out. Name variables corresponding to the question and section in the
questionnaire – include a dictionary Code descriptive answers (to facilitate analysis) All fields should be filled (NA or NR) Units should be uniform by district Totals calculated by formula not from summary column
Consistency checks – check for missing entries, wrong entries, sample statistics, patterns (queries should be inbuilt)
Validity checks – similar questions in different places on the questionnaire (RCH example)
![Page 22: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/22.jpg)
P3: Check-list (cont’d)
Data analysis Common mistakes in interpreting data
No analysis! No correlations, crosstabs, statistical significance levels or
regressions
Over generalizing the results Mis-reporting statistics Using % when the numbers are small Attributing causality when it is not demonstrated
![Page 23: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/23.jpg)
Phase 4: Report Writing
![Page 24: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/24.jpg)
P4: What the report should be … Simple, Clear and Relevant State limitations (attribution, causality) Major findings should be upfront Focus on quality rather than quantity Technical details in an appendix Should always
include the questionnaire in the appendix ask for electronic copy of data Request copies of filled out surveys
Essential if you change consultants at midterm or want to conduct internal analysis to compare modes of delivery (data lost example).
![Page 25: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/25.jpg)
How to manage the common errors Phase 1: Design
Clear objectives and hypotheses – know what you want to test
Identify a person in your unit who will manage this process
Write a good TOR, remember the baseline determines the quality of your panel
You can add questions as project evolves but cannot change questionnaire – threat to internal validity
Identify consultants Procurement – focus on quality not the cheapest bid “if you
throw peanuts you’ll attract monkeys” Ideally you should have a black-list of organizations
![Page 26: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/26.jpg)
How to manage the common errors
Phase 2: ImplementationOrganize an impact evaluation workshop if
necessaryRandomly verify questionnaires to reduce the
likelihood of false responses (no filling it in a bar)
Pay reasonable wages to surveyors (if possible)
Show the client and firm that you care
![Page 27: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/27.jpg)
How to manage the common errors
Phase 3: Data entry and analysis Double-data entry (2 separate organizations and
verify. Payment based on quality of data entry) Select 15 questionnaires at random and check data
entry – person in your unit managing Check data quality (consistency and validity checks) Hold an IE workshop to build data analysis capacity (if
necessary)
![Page 28: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/28.jpg)
How to manage the common errors
Phase 4: Report writingAgree on an outline beforehandDedicate a chapter on indicators you are
trackingFocus on quality not quantityThink “Big Picture”
![Page 29: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/29.jpg)
Elements of a Baseline Survey
Terms of References 1. Background: Project objectives and components2. Survey design: Consult a sampling expert!!!3. Survey instruments4. Guidance on survey implementation5. Data processing and analysis6. Staffing7. Duration and time schedule8. Submission of reports and datasets9. Support to the firm10. Budget & Payment Schedule11. Annexes: Draft questionnaires, Results Framework
![Page 30: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/30.jpg)
Baseline Survey Design: Typical Tasks for Consultants Recommend the methodology for sampling Calculate the optimal sample size Develop the sample frame and select the sample The final sample and details of the statistical
methodology used to select the sample need to be cleared by the project
Construct the sample weights and provide documentation on the methodology used to construct the weights
![Page 31: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/31.jpg)
Survey Instruments: Questionnaires Design or refinement and adaptation of the data
collection instruments Specify levels of data collection Length of questionnaires Prepare all support documentation including coding
guides, interviewer and supervisor manuals and the data entry manual
Translation and back-translation Skip patterns, coding open ended questions
![Page 32: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/32.jpg)
Guidance on Survey Implementation Implementation plan Selection and training of field workers: specify
minimum duration of training Pilot testing should be explicitly specified in ToR Responsibility for all field operations, including
logistical arrangements for data collection and obtaining household consent lies with Consultants.
Ask for field-work progress reports (bi-weekly/monthly)
![Page 33: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/33.jpg)
Staffing
Sampling expert/statistician Technical specialists as relevant Economist Sociologist. Core survey staff: the survey manager, the
field manager, the data manager Enumerators, supervisors and data entry
staff
![Page 34: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/34.jpg)
Baseline Report & Data
Explicitly request final electronic datasets—with complete documentation.
Agree on outline of baseline report up-front.
![Page 35: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/35.jpg)
Managing a Baseline Survey
Consult the experts—survey specialist and sampling specialist and develop the ToR in consultation.
Selection committee should include a survey expert and social scientists in addition to technical experts.
You can never over-supervise!!! Hire third-party supervision consultant if needed.
Question the data and the findings.
![Page 36: Developing a High Quality Baseline Salimah Samji & Mona Sur World Bank, New Delhi June 21, 2006](https://reader035.vdocuments.net/reader035/viewer/2022062417/5514fbb7550346935c8b61bb/html5/thumbnails/36.jpg)
Lets recap what you have learned
The devil lies in the detail Be watchful No pain, No gain