sampling for beneficiary-based surveys in support of food ... · sampling for beneficiary-based...

Sampling for Beneficiary-based Surveys in Support of Food for Peace Annual Monitoring Indicators

Presenter: Diana Stukel, FHI 360/FANTAFFP Annual Monitoring Workshop, September 11-15, 2017, Washington DC

POLL #1

Have you ever designed or implemented a survey with a statistically representative sample?

YES

NO

POLL #2

Have you ever calculated a sample size for a sample survey?

YES

NO

Introduction

Feed the Future (including Food for Peace (FFP)) Implementing Partners (IPs) must report annually to USAID on a set of annual monitoring indicators

Data collection options for annual monitoring

4

Routine monitoring

(RM)

Beneficiary-based surveys

(BBSs)

IntroductionFANTA Sampling Guide for Beneficiary-based Surveys and accompanying FANTA Excel-based Sample Size Calculator

(https://agrilinks.org/library/sampling-guide-beneficiary-based-surveys-select-feed-future-agricultural-annual-monitoring)

5

IntroductionCHAPTERS IN THE FANTA SAMPLING GUIDE FOR BBSs

Chapter 1 - Purpose and backgroundChapter 2 - Four selected Feed the Future annual monitoring

indicatorsChapter 3 - Comparison of RM and BBSsChapter 4 - When are BBSs appropriate?Chapter 5 - Timing and frequency of BBS data collectionChapter 6 - Issues to consider when outsourcing work to an external

contractorChapter 7 - Sampling frame guidance for BBSsChapter 8 - Overview of various approaches for collecting annual

monitoring data using BBSsChapter 9 - The Household Survey Approach (Approach 1)Chapter 10 - The Farmer Groups Approach (Approach 2)Chapter 11 - Sample weightingChapter 12 - Producing estimates of indicatorsChapter 13 - Producing confidence intervals and standard errors

associated with the indicators

Session Objectives

1. Learn under what circumstances BBSs are appropriate

2. Discuss a few approaches and sample design elements covered in the guide

3. Understand how to undertake a sample size calculation and observe a demonstration of the on-line Excel-based sample size calculator that accompanies the guide

7

Annual Monitoring Indicators• Annual monitoring indicators reported to Food for Peace span various

sectors: Agriculture, Maternal and Child Health and Nutrition (MCHN), Disaster Risk Reduction (DRR) /Resilience and Gender.

• Different data collection mechanisms may be needed to support indicators from different sectors

• Most Food for Peace annual monitoring indicators are agriculture-based indicators, so guide focuses on data collection for these

• Four agriculture-based annual monitoring indicators identified for which data collection is deemed particularly challenging

• All 4 indicators are either totals or functions of totals – so specific considerations for indicators of totals covered by the guide

• However, implementing partners also track project-specific/ custom indicators of proportions or means; sampling guide doesn’t cover this case, but this session will!!

Four Challenging FFP Agriculture-based Annual Monitoring Indicators (totals)

• Gross Margins

• Value of Incremental Sales

• Number of Farmers and Others who have Applied Improved Technologies

• Number of Hectares of Land under Improved Technologies

9

• Crop Yield: average production per hectare (mean)

• Average Coping Strategy Index of beneficiary households (mean)

• Mean decision making score for woman in beneficiary household (mean)

• Percentage of beneficiary farmers who used at least one improved storage technique (proportion)

• Percentage of beneficiary households reporting receiving risk and early warning information (proportion)

• Percentage of beneficiaries who know a neighbor or friend who has experienced domestic violence in the last month (proportion) 10

Examples of Project-Specific/ Custom Annual Monitoring Indicators

(means and proportions)

BBS versus Routine Monitoring?

Routine Monitoring data collection:• IMPLEMENTED BY: Project staff (M&E personnel, agriculture extension

workers, community health workers)

• LINKED WITH PROJECT IMPLEMENTATION: Data collection either concurrent with project implementation (e.g., during farmer field school)

• FREQUENCY OF COLLECTION: On a continuous basis throughout the year

• TARGET: ALL direct beneficiaries

11

BBS versus Routine Monitoring?

BBS data collection:• IMPLEMENTED BY: Third party firm typically (but can also be

implemented by project staff in some cases)

• GENERALLY NOT LINKED WITH PROJECT IMPLEMENTATION:Data collection not linked if BBS uses household survey approach, but can be linked if surveying farmer groups

• FREQUENCY OF COLLECTION: One or more data collections during year

• TARGET: A sample of direct beneficiaries

12

POLL #3What are the advantages of conducting a BBS over using Routine Monitoring (RM)? (Tick all that apply)Number of beneficiaries on which data collected is smaller

through BBSs than through RMBBSs allow for direct measurement on key data points

through visit to farmers’ plots (for agriculture surveys)BBSs do not require specialized skills in survey design and

implementationData from BBSs can be fed back to project staff more

frequently than data from RMData collection for BBSs can be integrated into project

implementation (e.g., at same time as farmer group meetings, for agriculture interventions)

Answers to PollWhat are the advantages of conducting a BBS over using Routine Monitoring (RM)? (Correct marked in green)Number of beneficiaries on which data collected is smaller through

BBSs than through RMBBSs allow for direct measurement on key data points through visit to

farmers’ plots (for agriculture surveys)BBSs do not require specialized skills in survey design and

implementationData from BBSs can be fed back to project staff more frequently than

data from RMData collection for BBSs can be integrated into project implementation

(e.g., at same time as farmer group meetings, for agriculture interventions)

14

When is it appropriate to conduct a BBS?

Large project and inadequate number of data collection staff

For agriculture data, farmer estimates of key agriculture-related data (such as plot size)

considered unreliable, and direct measurement through visit to farmer’s plot preferred

Lack of direct contact between

project and its beneficiaries

What is a Sampling Frame?

• Back-bone of BBSs!• Comprises complete lists of units for various stages at

which sampling takes place• E.g. Two Stage Cluster Sampling Design

• Frame 1: Complete list of clusters (e.g., villages or communities) in project implementation area

• Frame 2: Complete list of direct beneficiaries within each cluster

• E.g. One Stage Sampling Design• Frame 1: Complete list of direct beneficiaries within project

implementation area

16

Characteristics of a High Quality Sampling Frame

Comprehensive(extensive

information on each entity on

frame)

Complete(information on all

relevant entities; no missing entities; no

duplicates)

Up-to-date(as current as possible)

Importance of high quality beneficiary registration system!!!

Two Suggested Approaches to BBSs for annual monitoring Indicators

Two Suggested Approaches to BBSs for Annual Monitoring Indicators

Approach 1: Household Survey Approach• Resembles traditional household survey but on direct beneficiary

population rather than entire population in project implementation area

• Various survey design options:

• One-stage sampling of direct beneficiaries OR

• Two-stage sampling: first stage sampling of clusters (villages or communities) followed by second stage sampling of direct beneficiaries within sampled villages/ communities

• Households not directly sampled as in more traditional approaches but beneficiaries interviewed at their households

• Can use approach for survey data collection relating to indicators on agriculture, MCHN, DRR/ resilience, gender

• Lot Quality Assurance Sampling (LQAS) not recommended for collecting annual monitoring data 19


Approach 2: Farmer Groups (FG) Approach (for agriculture data only, but can be extended to data on MCHN, DRR/resilience, gender, etc.)

• Less traditional approach where interviewing conducted during project implementation (e.g., at a meeting of a FG during farmer field school)

• One survey design option (Two-stage sampling):

• First stage sampling of FGs (not households!)

• Second stage selection of ALL (not subset of!) direct beneficiary farmers within sampled FGs

• Farmers interviewed at next meeting of FG

• Approach not used for non-agriculture data, but could be extended to collect data on MCHN (for example), if project implementing Mother Care Groups 20


Approach 2: Farmer Groups Approach (for agriculture surveys only)

Appropriate:• For projects with large number of beneficiaries and

inadequate number of data collection staffAND • When direct measurement on key indicators (such as plot

size) NOT deemed necessary AND• When project works with farmer groups

21


Approach 2: Farmer Groups Approach (agriculture surveys only)

• Data relating to 4 agriculture indicators are collected with reliance on farmer recall

• No direct measurement of data (e.g., plot size) possible since no visit to farmer plot

• Data are sample weighted to represent all project beneficiaries receiving agriculture goods, services or training

• Less time & resource intensive than Approach 1, (no need to locate, visit and take measurements at households) but may yield less accurate results

22


Approach 2: Farmer Groups Approach (agriculture surveys only)

Challenges in implementing this approach:• Importance of maintaining frame of FGs and frame of

beneficiaries within each FG• Importance of interviewing farmers individually to avoid biased

results and to uphold confidentiality protection• Practical issues with interviewing all 15-30 farmers in a

sampled FG during one session; need for dedicated meeting and sufficient number of interviewers

• Unpredictable sample size since approach reliant on farmers attending next meeting of FG

23

24

How to Choose the Right Approach? (Agriculture Surveys Only)

POLL #4

Case Study Example: You are implementing a small project where you work directly with beneficiary farmers on a number of agriculture-based interventions. Experience in other surveys indicates that farmers provide inaccurate estimates of their plot sizes. Project interventions currently do not include periodic visits to farmers’ plots by M&E staff or agriculture extension workers. Using the flow diagram, what would be the best means of collecting annual monitoring data?

BBS using Household Survey ApproachBBS using Farmer Groups ApproachRoutine Monitoring (no survey)

Survey Steps for Approaches 1 & 2Regardless of approach used, BBS entail a number of survey design steps…..

26

Survey Design Steps for Drawing a Sample Using Approaches 1 & 2

Choose a survey design

option

Calculate the sample size

Choose the number of clusters to

select

Select a sample of clusters

Select survey respondents

One-stage design

Two-stage clustereddesign

Survey Step: Calculate the Sample Size

n=∑µα√4

• BBSs are “descriptive surveys” where the interest is in producing estimates of indicators to provide snapshot of situation at a single point in time• Aim is to achieve estimates with high “precision” (i.e., low standard errors)

by controlling the sample size

• Specific sample size formula needed for “single point in time” estimates

• Most Food for Peace annual monitoring indicators are totals or functions of totals• For these indicators, use specific sample size formula for “single point in

time” estimates of totals

• But often, Implementing Partners also monitor project-specific or custom indicators that are proportions or means• For these indicators, use specific sample size formula for “single point in

time” estimates of proportions or means 29

Calculate the Sample Size

POLL #5We do not use sample size formulas to support statistical tests of differences over time for indicators of totals. Why? (Check one only)

A) Testing for statistical differences of indicators of totals does not make sense

B) Testing differences of totals is not the main aim in the context of annual monitoring

• Gross Margins (function of totals)• Sample size formula complex, so not recommended to use

Gross Margins for sample size computation

• Value of Incremental Sales (total – constant)

• Number of Farmers and Others who have Applied Improved Technologies (total)

• Number of Hectares of Land under Improved Technologies (total)

31

Calculate the Sample Size:Four Challenging FFP Agriculture

based Annual Monitoring Indicators (totals)

Sample size formula for a single point-in-time estimator of a total:

𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔𝒊𝒊𝒔𝒔 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔 = 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 =𝑵𝑵𝟐𝟐 ∗ 𝒔𝒔𝟐𝟐 ∗ 𝒔𝒔𝟐𝟐

𝑴𝑴𝑴𝑴𝑴𝑴𝟐𝟐where

• N = total number of beneficiary farmers (if dealing with 4 agriculture indicators)

• z= critical value of normal distribution (typically use z= 1.96)

• s= standard deviation of the distribution of beneficiary data

• If no prior value available for s, use estimate:

s =𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑣𝑣𝑚𝑚𝑣𝑣𝑚𝑚𝑣𝑣 𝑜𝑜𝑜𝑜 𝑚𝑚𝑖𝑖𝑖𝑖𝑚𝑚𝑖𝑖𝑚𝑚𝑖𝑖𝑜𝑜𝑖𝑖 𝑜𝑜𝑜𝑜𝑖𝑖 𝑚𝑚𝑖𝑖𝑖𝑖𝑚𝑚𝑣𝑣𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑣𝑣 𝑏𝑏𝑣𝑣𝑖𝑖𝑣𝑣𝑜𝑜𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑖𝑖𝑏𝑏 − 𝑚𝑚𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑣𝑣𝑚𝑚𝑣𝑣𝑚𝑚𝑣𝑣 𝑜𝑜𝑜𝑜 𝑚𝑚𝑖𝑖𝑖𝑖𝑚𝑚𝑖𝑖𝑚𝑚𝑖𝑖𝑜𝑜𝑖𝑖 𝑜𝑜𝑜𝑜𝑖𝑖 𝑚𝑚𝑖𝑖𝑖𝑖𝑚𝑚𝑣𝑣𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑣𝑣 𝑏𝑏𝑣𝑣𝑖𝑖𝑣𝑣𝑜𝑜𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑖𝑖𝑏𝑏

6

• MOE = margin of error = p* target value of indicator

• p=acceptable percentage error; typically 0.05≤p≤0.1

• p=0.1 recommended for annual monitoring32

Calculate the Sample Size: Indicators of Totals

You are the M&E specialist for a Food for Peace project in Burundi. You have decided to conduct a beneficiary-based survey (BBS) to collect your agricultural annual monitoring data and you need to calculate the sample size for the survey. You have decided to use a household survey approach using a two-stage clustered design. One of the indicators of interest is the Value of Incremental Sales and you want to compute the sample size based on that indicator. Here is some information on your project:

• Number of beneficiaries: 60,000 small-holder farmers• Expected maximum value of sales for any individual beneficiary: US$1,200• Target value of sales over all beneficiaries for this year: US$15,000,000 (or

US$250 per beneficiary)

33

Case Study Example

Example: Value of Incremental Sales

• N=60,000 small-holder beneficiary farmers

• z= 1.96

• s: no prior value available so must use estimate

• Maximum value of sales for individual beneficiary is US$1,200

• Minimum value of sales for individual beneficiary is US$0

• s = (US$1,200 – US$0)/ 6 = US$200

34

Calculate the Sample SizeIndicators of Totals

Example: Value of Incremental Sales (continued)


• p=0.1 (recommended value)

• Target value of indicator is US$250 for each individual beneficiary, or US$15,000,000 for N=60,000 farmers

• MOE = 0.1*US$15,000,000 = US$1,500,000

• 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔𝒊𝒊𝒔𝒔 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔 = 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 =𝑁𝑁2∗𝑧𝑧2∗𝑠𝑠2

𝑀𝑀𝑀𝑀𝑀𝑀2= 60,000

2∗1.962∗2002

1,500,0002=246

35

Calculate the Sample Size:Indicators of Totals

Screen shot from accompanying sample size calculator: Indicators of Totals

SAMPLE SIZE CALCULATOR FOR BENEFICIARY-BASED SURVEYS IN SUPPORT OF SELECT FEED THE FUTURE AGRICULTURAL ANNUAL MONITORING INDICATORS

Please fill in appropriate values or select choices from the drop-down boxes provided for all cells highlighted in red.

SURVEY DESIGN OPTION 1 - Two-stage cluster design with systematic selection of beneficiaries

INDICATOR 2 - Value of incremental sales (collected at farm level) attributed to USG implementation

INITIAL SAMPLE SIZE N Population of beneficiaries 60,000

Estimate of standard deviation available? NO

If YES, write estimate here (in units of indicator):

If NO, provide estimates of minimum and maximum:

max Estimate of maximum (per beneficiary) 1,200.00

units of currency

min Estimate of minimum (per beneficiary) 0.00

units of currency

s Standard deviation 200.00

p Acceptable percentage error (Margin of error parameter) 10.0%

Target value of indicator (Margin of error parameter) 15,000,000.00

MOE Margin of error 1,500,000

Confidence level 95%

z Critical value from Normal Probability Distribution 1.96

ninitial Initial sample size 246

• Three adjustments to the initial sample size needed

• 𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔𝒊𝒊𝒔𝒔 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔 = 𝒊𝒊𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 = 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 ∗ 𝒊𝒊𝒂𝒂𝒂𝒂FPC ∗ 𝒊𝒊𝒂𝒂𝒂𝒂DEFF ∗ 𝒊𝒊𝒂𝒂𝒂𝒂𝑵𝑵𝑵𝑵

• 𝒊𝒊𝒂𝒂𝒂𝒂FPC = Finite Population Correction adjustment• Need for populations that are small relative to initial sample size (i.e., when 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊

> 𝟎𝟎.𝟎𝟎𝟎𝟎 ∗ 𝑵𝑵 )• Can reduce sample size somewhat in this case• Most often not needed and if not 𝒊𝒊𝒂𝒂𝒂𝒂FPC =1

• 𝒊𝒊𝒂𝒂𝒂𝒂DEFF = Design Effect (DEFF) adjustment• Need for two-stage cluster designs• Need to increase sample size to compensate for intra-cluster correlations• Use value of DEFF for same or similar indicators from prior surveys• In absence of prior information, use 𝒊𝒊𝒂𝒂𝒂𝒂DEFF =2

• 𝒊𝒊𝒂𝒂𝒂𝒂𝑵𝑵𝑵𝑵= Individual non-response adjustment• Need to increase sample size to compensate for anticipated non-response at

beneficiary level• Use values from prior similar surveys in same geographic area • In absence of prior information, assume 5% non-response or 𝒊𝒊𝒂𝒂𝒂𝒂NR = 1/(1-0.05)

38


Example: Value of Incremental Sales (continued)

• 𝒊𝒊𝒂𝒂𝒂𝒂FPC = Finite Population Correction adjustment

• Check: 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 = 246 is not greater than 𝟎𝟎.𝟎𝟎𝟎𝟎 ∗ 𝑵𝑵 = 𝟎𝟎.𝟎𝟎𝟎𝟎 ∗ 𝟔𝟔𝟎𝟎,𝟎𝟎𝟎𝟎𝟎𝟎 =𝟑𝟑,𝟎𝟎𝟎𝟎𝟎𝟎

• Adjustment not needed so 𝑚𝑚𝑖𝑖𝑎𝑎FPC =1

• 𝒊𝒊𝒂𝒂𝒂𝒂DEFF = Design Effect (DEFF) adjustment

• Assume absence of prior information, so 𝑚𝑚𝑖𝑖𝑎𝑎DEFF =2

• 𝒊𝒊𝒂𝒂𝒂𝒂𝑵𝑵𝑵𝑵= Non-response adjustment

• Assume absence of prior information, so 𝑚𝑚𝑖𝑖𝑎𝑎NR = 1/(1-0.05)

• 𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔𝒊𝒊𝒔𝒔 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔 = 𝒊𝒊𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 = 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 ∗ 𝒊𝒊𝒂𝒂𝒂𝒂FPC ∗ 𝒊𝒊𝒂𝒂𝒂𝒂DEFF ∗ 𝒊𝒊𝒂𝒂𝒂𝒂𝑵𝑵𝑵𝑵

= 𝟐𝟐𝟐𝟐𝟔𝟔 ∗ 𝟏𝟏 ∗ 𝟐𝟐 ∗𝟏𝟏

𝟏𝟏 − 𝟎𝟎.𝟎𝟎𝟎𝟎 = 𝟎𝟎𝟏𝟏𝟓𝟓

39


Screen shot from accompanying sample size calculator:Indicators of Totals

ADJUSTMENT 1 Use adjustment 1? (YES for all survey design options but only if n_initial /N is greater than 5%)NO

Finite population correction

ninitial / N Ratio of initial sample size to population size (%) 0.4%

nadj1 Adjusted sample size (1) 246

ADJUSTMENT 2 Use adjustment 2? (YES for survey design options 1,2 and 4) YES

Design effect D Design effect Enter value below:

2.00


ADJUSTMENT 3 Use adjustment 3? (YES for all survey design options) YES

Non-response Non-response rate 5.00%


FINAL SAMPLE SIZE nfinal Final sample size 518

DEMONSTRATION OF EXCEL-BASED SAMPLE SIZE CALCULATOR FOR INDICATORS OF TOTALS!!

41

Instructions


This Excel sample size calculator is designed to be used as a companion to the publication: "Sampling Guide for Beneficiary-Based Surveys for Select Feed the Future Agricultural Annual Monitoring Indicators" (see hyperlink below). The calculator was developed by Diana Stukel (FANTA/ FHI 360) and funded by USAID's Bureau for Food Security and Office of Food for Peace. It permits the calculation of sample sizes for beneficiary-based surveys that collect data for indicators that are estimates of totals—such as the USAID Feed the Future and Food for Peace annual monitoring indicators: "Number of Hectares under Improved Technologies"; "Value of Incremental Sales"; and the "Number of Farmers using Improved Technologies." The calculator requires a number of input parameters, as indicated in the instructions below, and produces both an initial sample size for a given specified indicator, as well as a final sample size after adjusting for the finite population correction, design effect, and non-response. See Sections 9.2.2 and 9.2.3 of the sampling guide for more details on the inputs to the formula used for the computations. The formula for the initial sample size can be found on page 39 and the formula for the final sample size can be found on page 46 of the sampling guide.

Click on this hyperlink to visit sampling guide

Instructions

For all cells highlighted in yellow, either fill in with appropriate values or select choices from the drop-down boxes provided.

The calculator contains 5 drop-down boxes that appear at the lower right hand corner of the cell when the yellow-highlighted cell is clicked. They contain the following preset choices:

A. Survey Design Option (See Sections 9.1.1, 9.1.2, 9.1.3 and 10.1 of the sampling guide for details on the preset choices)

1 - Two-stage cluster design with systematic selection of beneficiaries

2 - Two-stage cluster design with a listing operation and with systematic selection of beneficiaries

3 - One stage design with systematic selection of beneficiaries

4 - Two-stage cluster design of farmer groups with "take all" selection of beneficiary farmers

B. Indicator (See Section 9.2.2 of the sampling guide for details on the preset choices)

1 - Number of hectares under improved technologies or management practices as a result of USG assistance

2 - Value of incremental sales (collected at farm level) attributed to USG implementation

3 - Number of farmers and others who have applied improved technologies or management practices as a result of USG assistance

4- Other

C. Estimate of standard deviation available? (See Section 9.2.2 of the sampling guide for details on the preset choices)

YES

NO

D. Acceptable percentage error: Margin of error parameter (See Section 9.2.2 of the sampling guide for details on the preset choices)

5.0%

5.5%

6.0%

6.5%

7.0%

7.5%

8.0%

8.5%

9.0%

9.5%

10.0%

E. Confidence level or critical value from the Normal Probability Distribution (See Section 9.2.2 of the sampling guide for details on the preset choices)

90%

95%

This calculator is made possible by the generous support of the American people through the support of the Office of Health, Infectious Diseases, and Nutrition, Bureau for Global Health, U.S. Agency for International Development (USAID), USAID Bureau for Food Security, and USAID Office of Food for Peace, under terms of Cooperative Agreement No. AID-OAA-A-12-00005, through the Food and Nutrition Technical Assistance III Project (FANTA), managed by FHI 360.

The contents are the responsibility of FHI 360 and do not necessarily reflect the views of USAID or the United States Government.

http://agrilinks.org/library/sampling-guide-beneficiary-based-surveys-support-data-collection-selected-feed-future

Sample Size Calculator


Please fill in appropriate values or select choices from the drop-down boxes provided for all cells highlighted in yellow.

SURVEY DESIGN OPTION4- Two-stage cluster design of farmer groups with "take all" selection of beneficiary farmers

SELECT CHOICE FROM DROP-DOWN BOX ABOVE

INDICATOR3 - Number of farmers and others who have applied improved technologies or management practices as a result of USG assistance

Indicator title:SELECT CHOICE FROM DROP-DOWN BOX ABOVE

INITIAL SAMPLE SIZENPopulation of beneficiaries60,000

Estimate of standard deviation available?YES SELECT CHOICE FROM DROP-DOWN BOX TO LEFT

If YES, write estimate here (in units of indicator):0.50

If NO, provide estimates of minimum and maximum:

maxEstimate of maximum (per beneficiary)1.00

beneficiaries

minEstimate of minimum (per beneficiary)0.00

beneficiaries

sStandard deviation0.50

pAcceptable percentage error (Margin of error parameter)7.5% SELECT CHOICE FROM DROP-DOWN BOX TO LEFT

Target value of indicator (Margin of error parameter)30,000.00

MOEMargin of error2,250

Confidence level95% SELECT CHOICE FROM DROP-DOWN BOX TO LEFT

zCritical value from Normal Probability Distribution1.96

ninitialInitial sample size683

ADJUSTMENT 1Use adjustment 1? (YES for all survey design options but only if n_initial /N is greater than 5%)NO

Finite population correctionninitial / NRatio of initial sample size to population size (%)1.1%

nadj1Adjusted sample size (1)683

ADJUSTMENT 2Use adjustment 2? (YES for survey design options 1,2 and 4)YES

Design effectDDesign effectEnter value below:

2.00

nadj2Adjusted sample size (2)1,366

ADJUSTMENT 3Use adjustment 3? (YES for all survey design options)YES

Non-responseNon-response rate5.00%

nadj3Adjusted sample size (3)1,438

FINAL SAMPLE SIZEnfinalFinal sample size1,438

1 - Two-stage cluster design with systematic selection of beneficiaries90%5.0%1 - Number of hectares under improved technologies or management practices as a result of USG assistanceYES%

This calculator is made possible by the generous support of the American people through the support of the Office of Health, Infectious Diseases, and Nutrition, Bureau for Global Health, U.S. Agency for International Development (USAID), USAID Bureau for Food Security, and USAID Office of Food for Peace, under terms of Cooperative Agreement No. AID-OAA-A-12-00005, through the Food and Nutrition Technical Assistance III Project (FANTA), managed by FHI 360.

The contents are the responsibility of FHI 360 and do not necessarily reflect the views of USAID or the United States Government.2 - Two-stage cluster design with a listing operation and with systematic selection of beneficiaries95%5.5%2 - Value of incremental sales (collected at farm level) attributed to USG implementationNOUnits of indicator

3 - One stage design with systematic selection of beneficiaries6.0%3 - Number of farmers and others who have applied improved technologies or management practices as a result of USG assistance

4- Two-stage cluster design of farmer groups with "take all" selection of beneficiary farmers6.5%4- Other

7.0%

7.5%

8.0%

8.5%

9.0%

9.5%

10.0%

• Crop Yield: average production per hectare (mean)

• Average Coping Strategy Index of beneficiary households (mean)

• Mean decision making score for woman in beneficiary household (mean)

• Percentage of beneficiary farmers who used at least one improved storage technique (proportion)

• Percentage of beneficiary households reporting receiving risk and early warning information (proportion)

• Percentage of beneficiaries who know a neighbor or friend who has experienced domestic violence in the last month (proportion) 42

Calculate the Sample Size:Examples of Project-Specific/ Custom

Annual Monitoring Indicators (means and proportions)

Sample size formula for a single point-in-time estimator of a mean:

𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔𝒊𝒊𝒔𝒔 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔 = 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 =𝒔𝒔𝟐𝟐 ∗ 𝒔𝒔𝟐𝟐



• s= standard deviation of the distribution of beneficiary data

• If no prior value available for s, use estimate:

s =𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑣𝑣𝑚𝑚𝑣𝑣𝑚𝑚𝑣𝑣 𝑜𝑜𝑜𝑜 𝑚𝑚𝑖𝑖𝑖𝑖𝑚𝑚𝑖𝑖𝑚𝑚𝑖𝑖𝑜𝑜𝑖𝑖 𝑜𝑜𝑜𝑜𝑖𝑖 𝑚𝑚𝑖𝑖𝑖𝑖𝑚𝑚𝑣𝑣𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑣𝑣 𝑏𝑏𝑣𝑣𝑖𝑖𝑣𝑣𝑜𝑜𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑖𝑖𝑏𝑏 − 𝑚𝑚𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑣𝑣𝑚𝑚𝑣𝑣𝑚𝑚𝑣𝑣 𝑜𝑜𝑜𝑜 𝑚𝑚𝑖𝑖𝑖𝑖𝑚𝑚𝑖𝑖𝑚𝑚𝑖𝑖𝑜𝑜𝑖𝑖 𝑜𝑜𝑜𝑜𝑖𝑖 𝑚𝑚𝑖𝑖𝑖𝑖𝑚𝑚𝑣𝑣𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑣𝑣 𝑏𝑏𝑣𝑣𝑖𝑖𝑣𝑣𝑜𝑜𝑚𝑚𝑖𝑖𝑚𝑚𝑚𝑚𝑖𝑖𝑏𝑏

6


• p=acceptable percentage error; typically 0.05≤p≤0.1

• p=0.1 recommended for annual monitoring

• As before,

𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔𝒊𝒊𝒔𝒔 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔 = 𝒊𝒊𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 = 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 ∗ 𝒊𝒊𝒂𝒂𝒂𝒂FPC ∗ 𝒊𝒊𝒂𝒂𝒂𝒂DEFF ∗ 𝒊𝒊𝒂𝒂𝒂𝒂𝑵𝑵𝑵𝑵 43

Calculate the Sample Size: Indicators of Means

Sample size formula for a single point-in-time estimator of a proportion:

𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔𝒊𝒊𝒔𝒔 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔 = 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 =𝒔𝒔𝟐𝟐 ∗ 𝑷𝑷 ∗ (𝟏𝟏 − 𝑷𝑷)



• P= estimate of proportion from prior survey

• MOE = margin of error = p• p=acceptable percentage error; typically 0.05≤p≤0.1

• p=0.1 recommended for annual monitoring

• As before,

𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔𝒊𝒊𝒔𝒔 𝒔𝒔𝒊𝒊𝒔𝒔𝒔𝒔 = 𝒊𝒊𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 = 𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 ∗ 𝒊𝒊𝒂𝒂𝒂𝒂FPC ∗ 𝒊𝒊𝒂𝒂𝒂𝒂DEFF ∗ 𝒊𝒊𝒂𝒂𝒂𝒂𝑵𝑵𝑵𝑵

44

Calculate the Sample Size: Indicators of Proportions

• Each indicator has own sample size affiliated with it, but we need to settle on overall sample size for survey from among these

• Any one survey should be limited to collecting data for indicators with same underlying sampling group (e.g., beneficiary farmers)

• Must conduct different surveys for different sampling groups – generally related to different multi-sectoral interventions (e.g., MCHN versus agriculture)

• To compute overall sample size for survey, don’t need to base it on all indicators related to sample group – just key indicators related to sampling group

• Can include both FFP indicators and project-specific/ custom indicators as part of set of key indicators

45

Calculate the Overall Sample Size for Survey

• Do sample size computation for key indicators belongingto sampling group (e.g., beneficiary farmers)

• Take largest (affordable) sample size among all sample sizes computed

• ”Number of Farmers and Others who have Applied Improved Technologies” is indicator with greatest sample size requirement, so overall sample size for survey is 809 46

Calculate the OverallSample Size for Survey

INDICATOR FINAL SAMPLE SIZE

Value of Incremental Sales (total) 518

Number of Farmers and Others who have Applied Improved Technologies (total)

809

Number of Hectares of Land under Improved Technologies (total)

360

Crop Yield per Hectare (mean) 562

Percentage of beneficiary farmers who used at least one improved storage technique (proportion)

202

It is recommended to use a minimum sample size of 525 beneficiaries (including inflation for non-response) for any BBS –so that the sample size after non-response is about 500 beneficiaries

Why?

• To ensure reasonable precision for Food for Peace required disaggregates

• To ensure reasonable precision for any district or other sub-project geographic areas for which IPs wish to produce estimates

• To justify base cost of survey47

Calculate the Overall Sample Size for Survey

Survey Design Steps for Drawing a Sample Using Approaches 1 & 2

Choose a survey design

option

Calculate the sample size

Choose the number of clusters to

select

Select a sample of clusters

Select survey respondents

One-stage design

Two-stage clustereddesign

After data collection, analysis…

• Data entry (if not using a tablet or PDA)• Data cleaning (consistency and logic checks)• Production of sampling weights to take into account:

• Probability of selection at each stage of sampling

• Non-response at individual beneficiary level

• Production of estimates of indicators (using software packages)

• Production of associated confidence intervals and standard errors (using software packages)

49

This presentation is made possible by the generous support of the American people through the support of the Office of Health, Infectious Diseases and Nutrition, Bureau for Global Health, United States Agency for International Development (USAID); the Bureau for Food Security; and the Office of Food for Peace, Bureau for Democracy, Conflict and Humanitarian Assistance, under terms of Cooperative Agreement No. AID-OAA-A-12-00005, through FANTA, managed by FHI 360. The contents are the responsibility of FHI 360 and do not necessarily reflect the views of USAID or the United States Government.

DISCARDED SLIDES

Example: Value of Incremental Sales

𝑉𝑉𝑉𝑉𝑉𝑉 = 𝑉𝑉𝑉𝑉𝑅𝑅𝑅𝑅 − [𝑁𝑁𝑅𝑅𝑅𝑅𝑁𝑁𝐵𝐵𝑅𝑅

𝑉𝑉𝑉𝑉𝐵𝐵𝑅𝑅]

where

• VSRY = Value of sales for reporting year (value obtained from current BBS)

• VSBY = Value of sales for base year (value is known from base year data collection)

• NRY = Number of beneficiaries for reporting year (value is known from project records)

• NBY = Number of beneficiaries for base year (value is known from base year project records)

• At time of BBS, all above values known constants except VSRY• VIS is essential a total minus a constant, i.e., a total

52

Calculate the Sample Size

Calculate the Sample Size: Four Challenging Agriculture-based Annual

Monitoring Indicators

# Farmers and Others who have Applied Improved Technologies:Direct beneficiaries throughout

the value chain

Gross Margins and Incremental Sales:

Crops, animals, fish; direct beneficiaries

Smallholder producers only!Smallholder

producers of crops

Smallholder producers of animals, fish

# Hectares Under Improved

Technologies:Land-based (crop) technologies only

All direct beneficiary producers

Larger crop producers

Producers

Others

Sampling for Beneficiary-based Surveys in Support of �Food for Peace Annual Monitoring Indicators �POLL #1POLL #2IntroductionIntroductionIntroductionSession ObjectivesAnnual Monitoring IndicatorsFour Challenging FFP Agriculture-based Annual Monitoring Indicators (totals)Examples of Project-Specific/ Custom Annual Monitoring Indicators �(means and proportions)BBS versus Routine Monitoring?BBS versus Routine Monitoring?POLL #3Answers to PollWhen is it appropriate to conduct a BBS?What is a Sampling Frame? Characteristics of a High Quality Sampling Frame Two Suggested Approaches to BBSs for annual monitoring IndicatorsTwo Suggested Approaches to BBSs for Annual Monitoring IndicatorsTwo Suggested Approaches to BBSs for Annual Monitoring IndicatorsTwo Suggested Approaches to BBSs for Annual Monitoring IndicatorsTwo Suggested Approaches to BBSs for Annual Monitoring IndicatorsTwo Suggested Approaches to BBSs for Annual Monitoring IndicatorsHow to Choose the Right Approach? (Agriculture Surveys Only)POLL #4Survey Steps for Approaches 1 & 2Survey Design Steps for Drawing a Sample Using Approaches 1 & 2Survey Step: �Calculate the Sample SizeCalculate the Sample SizePOLL #5Calculate the Sample Size:�Four Challenging FFP Agriculture �based Annual Monitoring Indicators (totals)Calculate the Sample Size: Indicators of TotalsCase Study ExampleCalculate the Sample Size�Indicators of TotalsCalculate the Sample Size:�Indicators of TotalsScreen shot from accompanying sample size calculator: �Indicators of TotalsSlide Number 37Calculate the Sample Size: �Indicators of TotalsCalculate the Sample Size: �Indicators of TotalsScreen shot from accompanying sample size calculator:�Indicators of TotalsSlide Number 41Calculate the Sample Size:�Examples of Project-Specific/ Custom Annual Monitoring Indicators �(means and proportions)Calculate the Sample Size: Indicators of MeansCalculate the Sample Size: Indicators of ProportionsCalculate the Overall �Sample Size for SurveyCalculate the Overall�Sample Size for SurveyCalculate the Overall �Sample Size for SurveySurvey Design Steps for Drawing a Sample Using Approaches 1 & 2After data collection, analysis…Slide Number 50DISCARDED SLIDESCalculate the Sample SizeCalculate the Sample Size: �Four Challenging Agriculture-based Annual Monitoring Indicators

sampling for beneficiary-based surveys in support of food ... · sampling for beneficiary-based...

Documents