approach to and findings from farming practices survey
DESCRIPTION
TRANSCRIPT
Approach To and Findings From Farming Practices Survey
October 17, 2008
Approach To and Findings From Farming Practices Survey
October 17, 2008
Overview of Presentation Overview of Presentation
I. How an Impact Evaluation Works
II. Evaluation Design for Water-to-Market Training
III. About the Farming Practices Survey
IV. Findings from the Farming Practices Survey
I. How an Impact Evaluation Works
II. Evaluation Design for Water-to-Market Training
III. About the Farming Practices Survey
IV. Findings from the Farming Practices Survey
Impact EvaluationImpact Evaluation
Was the program effective?
Estimating impacts involves comparing: – Outcomes with the program with outcomes if there
were no program
Counterfactual: What participants would have experienced if there were no program– True counterfactual is not directly observed
Goal of impact study is to identify a comparison group to approximate the counterfactual
Was the program effective?
Estimating impacts involves comparing: – Outcomes with the program with outcomes if there
were no program
Counterfactual: What participants would have experienced if there were no program– True counterfactual is not directly observed
Goal of impact study is to identify a comparison group to approximate the counterfactual
Importance of the Counterfactual:An Illustrative Example
Importance of the Counterfactual:An Illustrative Example
Rural poverty has decreased in recent years, prior to MCA-Armenia programs
In this case, we want to see how much more did poverty decline because of MCA-Armenia?
Rural poverty has decreased in recent years, prior to MCA-Armenia programs
In this case, we want to see how much more did poverty decline because of MCA-Armenia?
How Much More Did Poverty Decline Because of MCA-Armenia?
How Much More Did Poverty Decline Because of MCA-Armenia?
Poverty With and Without MCA
2005 2006 2007 2008 2009 2010
Po
ve
rty
Ra
te
Impact
Random Assignment: Often Considered the “Gold Standard”
Random Assignment: Often Considered the “Gold Standard”
Randomly assign into two groups, similar to a lottery
Program and comparison groups are the same on average, except one group has access to program – Subsequent differences in outcomes can be attributed
to the program
Random assignment most feasible when there are limited resources/excess demand
Randomly assign into two groups, similar to a lottery
Program and comparison groups are the same on average, except one group has access to program – Subsequent differences in outcomes can be attributed
to the program
Random assignment most feasible when there are limited resources/excess demand
II. Evaluation for Water-to-Market Training
II. Evaluation for Water-to-Market Training
Water-to-Market ActivitiesWater-to-Market Activities
Focus on evaluation of training programs– Water management techniques– Transition to higher value agriculture
ACDI/VOCA providing training – Primarily in Compact Years 2-5
Focus on evaluation of training programs– Water management techniques– Transition to higher value agriculture
ACDI/VOCA providing training – Primarily in Compact Years 2-5
Evaluation Design for Water-to-Market Training
Evaluation Design for Water-to-Market Training
Ideally, would want to randomly assign farmers who apply for training
Not practically feasible or politically viable– New program starting, so no excess demand
– Intervention at the village level (potential for
spillover/contamination)
We randomly assigned villages to program or “control” group– Use a phased-in approach
Ideally, would want to randomly assign farmers who apply for training
Not practically feasible or politically viable– New program starting, so no excess demand
– Intervention at the village level (potential for
spillover/contamination)
We randomly assigned villages to program or “control” group– Use a phased-in approach
Shortcomings of the Approach Shortcomings of the Approach
Less statistical power to detect impacts than random assignment at the individual level – Not all farmers in a village may want to participate in
training – Clustering effects (village specific effects)
Challenge can be overcome to some extent if:– Take-up rates of training high – More villages included
Less statistical power to detect impacts than random assignment at the individual level – Not all farmers in a village may want to participate in
training – Clustering effects (village specific effects)
Challenge can be overcome to some extent if:– Take-up rates of training high – More villages included
Random Assignment Design Random Assignment Design
Randomly assign when training will start in each village, with three groups:– Compact Year 2 – Compact Years 3 and 4– Compact Year 5
Compare Compact Year 2 villages to Compact Year 5 villages
Randomly assign when training will start in each village, with three groups:– Compact Year 2 – Compact Years 3 and 4– Compact Year 5
Compare Compact Year 2 villages to Compact Year 5 villages
Random Selection of Villages (Implemented in August 2007) Random Selection of Villages
(Implemented in August 2007)
Started with a list of villages with “good water” and which could benefit from training – Excluded villages served in pilot phase
Random selection conducted publicly to: – Ensure transparency – Allow for greater accountability and
implementation fidelity
Started with a list of villages with “good water” and which could benefit from training – Excluded villages served in pilot phase
Random selection conducted publicly to: – Ensure transparency – Allow for greater accountability and
implementation fidelity
Selection of Villages Selection of Villages
Total of 277 village clusters were assigned– Year 2: 120 clusters– Years 3 and 4: 77 clusters– Year 5: 80 clusters
Selection of clusters stratified by WUAs– In proportion to the number of villages in the WUA
To ensure balance across treatment and control villages – Also for political reasons
Total of 277 village clusters were assigned– Year 2: 120 clusters– Years 3 and 4: 77 clusters– Year 5: 80 clusters
Selection of clusters stratified by WUAs– In proportion to the number of villages in the WUA
To ensure balance across treatment and control villages – Also for political reasons
Selection of Villages (cont’d) Selection of Villages (cont’d)
No. of Villages By Year of Training and Ag. Zone No. of Villages By Year of Training and Ag. Zone All Zones Ararat Valley Pre-Mountainous Mountainous Subtropical
Year 2: treatment 120 44 58 12 6Years 3 and 4: nonresearch 77 18 19 38 2Year 5: control 80 28 38 10 4Total 277 90 115 60 12
Overall Approach to the Impact Evaluation
Overall Approach to the Impact Evaluation
Compare outcomes for farmers in treatment and control villages
Examine impacts on key outcomes– Participation in training and adoption of new practices– Changing crop patterns, improved yields, and
increases in income
Data and Sample – Survey of farmers in the treatment and control
villages– Sample of 5,000 farmers to detect impact on poverty
of 5 percentage points
Compare outcomes for farmers in treatment and control villages
Examine impacts on key outcomes– Participation in training and adoption of new practices– Changing crop patterns, improved yields, and
increases in income
Data and Sample – Survey of farmers in the treatment and control
villages– Sample of 5,000 farmers to detect impact on poverty
of 5 percentage points
About the Farming Practices Survey
About the Farming Practices Survey
Farming Practices Survey Farming Practices Survey
Baseline survey conducted by AREG
Implemented during Nov 2007- Feb 2008 – In 223 communities– Target sample of 5,000 farmers
Topics covered in the survey – Demographic characteristics– Agricultural practices and productivity – Income and consumption
Baseline survey conducted by AREG
Implemented during Nov 2007- Feb 2008 – In 223 communities– Target sample of 5,000 farmers
Topics covered in the survey – Demographic characteristics– Agricultural practices and productivity – Income and consumption
Identifying the Sample Frame Identifying the Sample Frame
Sample frame is critical as it defines what population the study represents – Need comparable frame for treatment and control
villages– Want persons who farm as their main occupation
No viable sample frame exists for the surveys – Originally attempted to use WUA member lists– Worked with WUA heads to get names of farmers – During pretest, AREG found some lists were bad – All lists had to be verified by AREG in the field and
developed as necessary
Sample frame is critical as it defines what population the study represents – Need comparable frame for treatment and control
villages– Want persons who farm as their main occupation
No viable sample frame exists for the surveys – Originally attempted to use WUA member lists– Worked with WUA heads to get names of farmers – During pretest, AREG found some lists were bad – All lists had to be verified by AREG in the field and
developed as necessary
Implementing the Survey and Checking Data Quality
Implementing the Survey and Checking Data Quality
Interactive interviewer training (Q by Q, etc)
Piloting of both survey instrument and sample list – Revisions to questionnaire– Sample verification process
Regular generation of field reports
MCA did field visits to review processes
Independent review of initial batches of data
Interactive interviewer training (Q by Q, etc)
Piloting of both survey instrument and sample list – Revisions to questionnaire– Sample verification process
Regular generation of field reports
MCA did field visits to review processes
Independent review of initial batches of data
Planned Improvements to the Upcoming Survey
Planned Improvements to the Upcoming Survey
More use of photos in training interviewers
on farming practices
Pretesting both survey instrument and sample list
Double key data entry
More use of photos in training interviewers
on farming practices
Pretesting both survey instrument and sample list
Double key data entry
Baseline FPS Survey Baseline FPS Survey
Completed interviews with 4,855 farmers
Sample focuses on households farming for five years or more in these communities
Sample not representative of all farmers in Armenia
However, sample internally valid for study– Will show comparison of farmers in treatment
and control villages later
Completed interviews with 4,855 farmers
Sample focuses on households farming for five years or more in these communities
Sample not representative of all farmers in Armenia
However, sample internally valid for study– Will show comparison of farmers in treatment
and control villages later
IV. Baseline FindingsIV. Baseline Findings
Farmer and Household Characteristics
Crop Cultivation and Sales
Income and Poverty
Farmer and Household Characteristics
Crop Cultivation and Sales
Income and Poverty
General Approach to Analysis and Data Cleaning
General Approach to Analysis and Data Cleaning
Descriptive analysis – Averages and distributions– Examined conditional and unconditional measures– Used weights to adjust for treatment-control balance
Data checks – Missing values and outliers – Logical skips – Internal consistency
Measuring income and consumption is challenging – Sensitivity tests – Looking at distributions critical
Descriptive analysis – Averages and distributions– Examined conditional and unconditional measures– Used weights to adjust for treatment-control balance
Data checks – Missing values and outliers – Logical skips – Internal consistency
Measuring income and consumption is challenging – Sensitivity tests – Looking at distributions critical
Farmer and Household Characteristics
Farmer and Household Characteristics
Farmer and Household Characteristics
Farmer and Household Characteristics
Characteristics (Percentages) Characteristics (Percentages)
<40 18.7 40–49 35.5 50–59 27.7 60 or older 18.1 (Average) (49.2)
Less than secondary 13.6 Secondary 40.8 Secondary (vocational) 28.0 More than secondary 17.6Female Respondent 11.9Household Size 5.3 Children 1.4
Respondent’s Age
Respondent’s Education
Farmer and Household Characteristics (cont’d)Farmer and Household Characteristics (cont’d)
Land Cultivated by Respondents (Sq. Meters) Land Cultivated by Respondents (Sq. Meters) All Zones Ararat Valley Pre-Mountainous Mountainous Subtropical
Average 19,846 14,274 22,445 30,776 23,638Median 10,000 7,000 14,000 20,000 13,000
Average 1,722 1,700 1,630 2,377 1,247Median 1,200 1,200 1,200 2,000 1,000
Area of Land Cultivated
Size of Kitchen Plot
Few Farmers Use Water Management Practices
Few Farmers Use Water Management Practices
Respondents’ Irrigation Practices (Percentages)Respondents’ Irrigation Practices (Percentages)
Furrow row spacing 7.4
Scientific scheduling 0.1
Water meters 0.0
Non-pressure/pipe irrigation 0.6
Pressure irrigation 0.3
Respondents Using:
Crop Cultivation and SalesCrop Cultivation and Sales
Crop Cultivation and SalesCrop Cultivation and Sales
Respondents Growing and Selling Crops (Overall Percentages)Respondents Growing and Selling Crops (Overall Percentages)
Respondents Growing
Respondents Selling
Grains 44.6 7.0Fruits 77.0 36.4Vegetables 32.4 16.5Herbs 5.1 2.1Potatoes 25.7 5.8Nuts 6.3 0.7Grass 38.1 6.6Other 13.8 4.7
Crop Cultivation and Sales Vary by Agricultural Zone
Crop Cultivation and Sales Vary by Agricultural Zone
• Farmers in Ararat Valley grow more fruit, and much more likely to sell their fruits and vegetables
• Farmers in the Mountainous Zone grow much more grain and potatoes
• Mountainous Zone farmers sell very little
• Farmers in Ararat Valley grow more fruit, and much more likely to sell their fruits and vegetables
• Farmers in the Mountainous Zone grow much more grain and potatoes
• Mountainous Zone farmers sell very little
Income and PovertyIncome and Poverty
How to Measure Agricultural Income?How to Measure Agricultural Income?
• Crop sales tell us monetary income
• Much of production is consumed by household; not reflected in sales
• Assign monetary value to crops consumed by household
• Subtract agricultural costs to get profit
• Crop sales tell us monetary income
• Much of production is consumed by household; not reflected in sales
• Assign monetary value to crops consumed by household
• Subtract agricultural costs to get profit
Crop Sales and Value of ProductionCrop Sales and Value of Production
Respondents’ Average Crop Sales and Values (AMD)Respondents’ Average Crop Sales and Values (AMD)Sales Value (Production x Price)
Grains 27,136 110,458Fruits 261,839 307,462Vegetables 73,214 126,970Herbs 9,585 12,020Potatoes 34,653 62,010Nuts 590 3,534Grass 18,745 63,852Other 23,785 32,702Total 449,822 707,133
Measuring IncomeMeasuring Income
• Survey is relatively short (30 minutes)
• Focus on agricultural production, but learn about non-agricultural income/consumption as well
• Income measures include employment income, pensions, remittances, other benefits
• Consumption measures include groceries, household products, utilities, transportation
• Survey is relatively short (30 minutes)
• Focus on agricultural production, but learn about non-agricultural income/consumption as well
• Income measures include employment income, pensions, remittances, other benefits
• Consumption measures include groceries, household products, utilities, transportation
Household Income, Only Considering Monetary Income
Household Income, Only Considering Monetary Income
Annual Household Monetary Income (AMD)Annual Household Monetary Income (AMD)
Monetary agricultural profit (crop sales – costs)
Total Monetary Income
344,000
165,983
818,520
461,930 300,000
627,913
Nonagricultural income
Monetary agricultural profit (crop sales – costs)
Total Monetary Income
-35,950
Method 1 (Monthly Salary x 12) MedianAverage
384,000652,538
-35,950
451,000
Nonagricultural income
Method 2 (Monthly Salary x 6) Average Median
165,983
Household Income, Including Value of Crops Consumed
Household Income, Including Value of Crops Consumed
Annual Household Economic Income (AMD)Annual Household Economic Income (AMD)
113,667
885,224 522,577
423,293
Method 1 (Monthly Salary x 12) Average
Total Economic Income
461,930
Total Economic Income
Nonagricultural income
Nonagricultural income
Economic agricultural profit (crop value – costs)
Economic agricultural profit (crop value – costs)
300,000
Method 2 (Monthly Salary x 6) Average
652,000
Median
652,538 384,000
423,293 113,667
Median
1,075,831
Measuring PovertyMeasuring Poverty
• Consumption-based measure
• Adopt similar approach as NSS/World Bank
• Food poverty line: Meets minimal caloric needs
• Complete poverty line: Allowance for other basic necessities
• Consumption-based measure
• Adopt similar approach as NSS/World Bank
• Food poverty line: Meets minimal caloric needs
• Complete poverty line: Allowance for other basic necessities
Estimates of Poverty RatesEstimates of Poverty Rates
Respondent Households in Poverty (Percentages)Respondent Households in Poverty (Percentages) Food
PovertyComplete Poverty
Excluding consumption of own crop production 11.5 26.1Including consumption of own crop production 7.5 18.3Average household consumption relative to poverty line 349 237
Poverty Varies Appreciably by ZonePoverty Varies Appreciably by Zone
Respondent Households in Poverty by Zone (Percentages) Respondent Households in Poverty by Zone (Percentages)
0
5
10
15
20
25
30
Ararat Valley Pre-Mountainous Mountainous Subtropical
Food Poverty
Complete Poverty
0
5
10
15
20
25
30
Ararat Valley Pre-Mountainous Mountainous Subtropical
Food Poverty
Complete Poverty
Many Households are Near Poverty Line
Many Households are Near Poverty Line
Consumption Relative to Complete Poverty Line (CPL) Consumption Relative to Complete Poverty Line (CPL)
Including own crop consumption
0
5
10
15
20
25
30
35
40
45
Below CPL 1-2 timesCPL
2-3 timesCPL
3-4 timesCPL
4-5 timesCPL
5 or moretimes CPL
Including own crop consumption
0
5
10
15
20
25
30
35
40
45
Below CPL 1-2 timesCPL
2-3 timesCPL
3-4 timesCPL
4-5 timesCPL
5 or moretimes CPL
Values of Variables byTreatment Status
Values of Variables byTreatment Status
Demographic Characteristics and Land Size of Sample Members
Demographic Characteristics and Land Size of Sample Members
Treatment Control
Respondent’s Age (years) 49.2 49.2 Female Respondent 13.0 10.8 Respondents Education
Less than Secondary 13.8 13.4Full Secondary 40.6 41.1Secondary Vocational 28.9 27.1More than Secondary 16.7 18.4
Area of land cultivated (ha) 1.99 1.98Kitchen plot size (ha) .171 .173
Use varying furrow spacing 7.5 7.4
Crop Cultivation and Sales and Income for Sample Members Crop Cultivation and Sales and Income for Sample Members
Treatment Control
Percent Cultivating Grain 45.7 43.5Fruit 76.3 77.6 Vegetable 33.1
31.7
Revenue from Crops Sold (AMD) Grain 21,711 32,531Fruit 317,619 206,355 Vegetable 88,858
57,653
Market Value of Harvest Grain 112,784 108,144 Fruit 332,365 282,692 Vegetable 110,924
142,928
Household Income and Poverty Household Income and Poverty
Treatment Control
Household Income (AMD) Nonagricultural Income 674,780 630,414 Total Monetary Income 929,592 708,041** Total Economic Income 1,115,105 1,036,766
PovertyComplete Poverty Rate 17.9% 18.7% Consumption Relative to CPL 238% 237%
Lessons Learned and Plans for the Future
Lessons Learned and Plans for the Future
Modifications to the questionnaire – More detailed measure of employment income– Consistent units for crop production
Enhancements to data collection procedures – Use pictures of irrigation practices – Focus on high response rates
More focus on longitudinal analysis – Useful to factor out preexisting differences
Final impact report (2011) – Possibility of interim report
Modifications to the questionnaire – More detailed measure of employment income– Consistent units for crop production
Enhancements to data collection procedures – Use pictures of irrigation practices – Focus on high response rates
More focus on longitudinal analysis – Useful to factor out preexisting differences
Final impact report (2011) – Possibility of interim report