improving market access of farmer groups in uganda: evaluating the role of working capital
TRANSCRIPT
Impact evaluation An application to farmers groups in Uganda
Improving market access of farmer groups inUganda: evaluating the role of working capital
Ruth Vargas Hill and Eduardo Maruyama
May 9, 2012
Impact evaluation An application to farmers groups in Uganda
Outline
Impact evaluationIntroductionRCTs
An application to farmers groups in UgandaIntroductionImplementationResultsConcluding remarks
Impact evaluation An application to farmers groups in Uganda
Why evaluate?
• Evaluating interventions (policies or programs) helps:• Understand the actual rather than the anticipated effects of
programs.• Determine how to design new programs.• Determine the most cost-effective approach to achieve a
desired goal.
Impact evaluation An application to farmers groups in Uganda
Estimating impact: introduction
• When we conduct impact evaluation we assess how a programaffects the well-being or welfare of individuals, households orcommunities:
• Profitability of agricultural production• Increased income or consumption (or other measures of
welfare) of rural households• Poverty levels or growth rates at the community level
Impact evaluation An application to farmers groups in Uganda
Impact evaluation versus other M&E tools
• Impact evaluation is different from other M&E tools in that itfocuses on discerning the impact of the program from allother confounding effects.
• The focus of impact evaluation is providing evidence of thecausal link between an intervention and an outcome.
• This is why impact evaluation is a powerful too, but also whatmakes it difficult to implement in practice.
Impact evaluation An application to farmers groups in Uganda
Impact evaluation versus other M&E tools
Inputs Outputs Outcomes Impacts
HighDIFFICULTY OF SHOWING CAUSALITYLow
Visits by extensions agents,
physical inputs (such as seeds)
Knowledge of the new
technology
Use of the new
technology
Increased yields, higher farm profits, improved
consumption
Example: A program of providing advice on a new technology to farmers
Impact evaluation An application to farmers groups in Uganda
Essential component: counterfactual
• Difficulty is determining what would have happened to theindividuals or communities of interest in absence of theproject.
• We are interested in the difference in an outcome for anindividual with and without the intervention.
• Problem: can only observe people in one state of the world atone time
• The key component to an impact evaluation is to construct asuitable comparison group to proxy for the counterfactual.
Impact evaluation An application to farmers groups in Uganda
Before and after comparisons
• Why not collect data on individuals before and afterintervention (the Reflexive)? Difference in income, etc, wouldbe due to project
• Problem: many things change over time, including the project• The country is growing and profits are rising. Is this due to the
program or would have occurred in absence of program?• This is particularly a problem for agricultural interventions:
many factors affect yield (weather, availability of inputs) andprices in a given year.
Impact evaluation An application to farmers groups in Uganda
Comparison groups
• Instead of using before/after comparisons, we need to usecomparison groups to proxy for the counterfactual
• Two core problems in finding suitable groups:• Programs are targeted
• Recipients receive intervention for particular reason
• Participation is voluntary• Individuals who participate differ in observable and
unobservable ways (selection bias)
• Hence, a comparison of participants and an arbitrary group ofnon-participants can lead to misleading or incorrect results
Impact evaluation An application to farmers groups in Uganda
Randomizing to create a true comparison group
• We need a comparison group that is as identical in observableand unobservable dimensions as possible, to those receivingthe program, and a comparison group that will not receivespillover benefits.
• Number of techniques:• Randomized control trials (the gold standard)• Careful matching techniques: IV, propensity score matching,
regression discontinuity design
Impact evaluation An application to farmers groups in Uganda
Randomized Control Trials (RCTs)
• In RCTs, participation in a policy (or usually eligibility toparticipate in a policy) is randomly assigned.
• This is done to ensure that the only difference between thosein and out of an intervention, is their participation, and as aresult any difference between participants andnon-participants can be attributed to the program alone.
• Because participation (treatment) is randomized, thenon-treatment outcomes between those that are not treatedand those that are treated is equal.
Impact evaluation An application to farmers groups in Uganda
Households or groups of households
Impact evaluation An application to farmers groups in Uganda
Households or groups of households
TT
T
T
TT
T
T
TT
T
T
C
C
C
C
CC
C
C
C
C
CC
C
C
C
Impact evaluation An application to farmers groups in Uganda
Does randomization create a true comparison group?
• We can test that they are equal by collecting data on the twogroups before the intervention and checking that the averagecharacteristics of the two groups are the same.
• For the treatment and control groups to be statistically equalyou need a large number of each. Cannot have one treatedhousehold and one control household.
• Means that you cannot use this method to answer questionsabout country policy changes (e.g. fiscal policy changes).
• There are stragegies that can be used to ensure that thetreatment and control groups are equal (e.g. stratification).
Impact evaluation An application to farmers groups in Uganda
How do we estimate impact by randomizing?
• Identify the outcome we are interested in (e.g. yields, amountof output marketed, price received)
• Estimate the average of the outcome in the treatment group.
• Estimate the average of the outcome in the control group.
• Calculate the difference of these averages and test to see ifthe two averages are significantly different from each other.
• Average Treatment Effect
• Note: it is just differences in the AVERAGE outcome that areestimated.
Impact evaluation An application to farmers groups in Uganda
Challenges to estimating impact
• Sometimes the effect of the program is small.
• Or there are many other factors affecting the outcome ofinterest that it is hard to see if a difference is statisticallydifferent between two groups.
• We try and control for this in two ways:• Include a large number of households in treatment and control.
This increases our power to detect a small effect.• Collect data on characteristics of the household that may
influence the outcome variable at baseline (including thepre-intervention outcome of interest)
Impact evaluation An application to farmers groups in Uganda
Other challenges to estimating impact
• Are we sure that the intervention had no impact on thecontrol group? Are there no spillover effects? (E.g. on prices)
• Was there any attrition as a result of the program that meanswe miss capturing some of the impact? For example didpeople migrate as a result of the program? If so, we will misscapturing the effect of the program on these people.
• We randomized to avoid selection bias, but some of it stillmay remain:
• Did everyone in the treatment group participate as expected?• Did anyone in the control group participate even if they were
not meant to?
Impact evaluation An application to farmers groups in Uganda
Selection bias
36
Target Population
Target Population
Not in evaluation
Not in evaluation
Evaluation Sample
Evaluation Sample
Random Assignment
Treatment group
Treatment group
Participants
No-ShowsNo-Shows
Control groupControl groupNon-
ParticipantsNon-
Participants
Cross-oversCross-overs
Impact evaluation An application to farmers groups in Uganda
Handling selection bias
• Intent to treat (ITT):• Average impact of program in practice: treats all noncompliars
as treated, and treats all crossovers as remaining in the control• Problem: power is reduced by noncompliance and does not
provide an idea of what the average impact of the program onthe treated is.
• Treatment on the treated (ToT):• Instruments for take-up with assignment: gives an idea of the
average impact of the program for a specific group
Impact evaluation An application to farmers groups in Uganda
Summary of advantages and disadvantages
• Powerful method to identify causal impact of a policy orprogram.
• Careful design is needed to ensure you are able to detectchanges
• Can be expensive: baseline and follow-up, a large number ofparticipants (especially if it is to be representative)
• Only valid in some circumstances: randomization over anumber of units needs to be possible.
• Provides information on the average outcome.
• Internally valid, repetition and a theory of change needed tomake predictions from results (external validity).
Impact evaluation An application to farmers groups in Uganda
External validity
• Tells us whether something worked in a specific context,understanding whether an intervention would work again in adifferent setting, external validity, is very difficult to know.
• Great benefit of few assumptions, comes with great cost”narrowness of scope” (Cartwright 2007).
• Any external validity involves some assumptions about theproject working in different conditions.
• Yet replicating a project is almost impossible, replicatingtriggers of mechanisms that produce the change is often morepossible.
• Result: we need to repeat impact evaluations and also have atheory of change to know what will work in the future, i.e. toreally learn.
• But better than an approach that is not internally valid.Predictions cannot be made from this either.
Impact evaluation An application to farmers groups in Uganda
Other concerns in learning from results
• General equilibrium effects.
• Corruption in implementing a large scale.
• Capacity to implement at a large scale.
• Overlap between new environment and old (example ofmedicine)
• Not automatic to go from experiments to learning and policyadvice.
Impact evaluation An application to farmers groups in Uganda
Another approach
• We need a theory of change that guides us in going fromone-off impact evaluation to general lessons. This meansmake assumptions and, ideally, use experiments to test andrefine these assumptions.
• When designing impact evaluation for this purpose, it oftenlooks quite different:
• Theory of change influences the design of the impactevaluation.
• Often identifying the differential impact of different treatments,rather than the impact of one treatment against baseline.
Impact evaluation An application to farmers groups in Uganda
Outline
Impact evaluationIntroductionRCTs
An application to farmers groups in UgandaIntroductionImplementationResultsConcluding remarks
Impact evaluation An application to farmers groups in Uganda
Introduction
• Smallholder agriculture in Sub-Saharan Africa is largelyexposed to pervasive market failures, translating into missedopportunities and sub-optimal economic behavior.
• These failures are often rooted in the importance ofeconomies of scale in procuring inputs and marketing produce.
• By engaging in markets collectively through a farmers group,smallholders can overcome economies of scale.
• Despite the renewed interest from governments and donoragencies in farmers groups as a means to overcome thesemarket failures, evidence shows that they have so far hadlimited success.
Impact evaluation An application to farmers groups in Uganda
Ugandan context
• The majority of Ugandan farmers sell their (unprocessed)produce at harvest time to itinerant traders at the farm-gate.
• Survey of farmers groups engaged in some form of outputmarketing revealed that:
• Farmers get a higher price when they sell collectively.• Yet few farmers sell through the marketing group of which
they are a member (only 47% make sales through group)• Farmers are less likely to sell collectively when they are
liquidity constrained and in need of emergency money.• Groups that offer cash on delivery of produce (rather than
payment some days later) have a higher proportion ofmembers selling through the group.
Impact evaluation An application to farmers groups in Uganda
Key impact question
• Would providing working capital loans to farmers groups sothat they can provide cash on delivery, improve marketingoutcomes for farmers?
• We cannot infer this from the baseline data: good groups maybe better at collective sales and better able to access financewhich allows payment on delivery.
• We would like to compare groups of similar quality and see ifworking capital loans increase sales amongst those thatreceived them.
Impact evaluation An application to farmers groups in Uganda
Testing a theory of change
• Farmer groups can offer higher prices but because of thewaiting times involved in receiving payment, farmers find itcostly to sell though the group.
• Farmers are liquidity constrained and often sell coffee to meeturgent financial needs, so even small delays in payment can beproblematic.
• Waiting for payment involves a high level of trust in the abilityof the group to market and transparency. There is a risk if thegroups cannot be trusted.
• Enabling groups to make payment on delivery through aworking capital loan will reduce the cost of selling through thegroup.
• More farmers will sell through the group and receive higherprices as a result.
Impact evaluation An application to farmers groups in Uganda
The impact of working capital credit• Randomized provision of working capital credit to farmers
groups that had already been engaged in output marketing:• Provide selected groups with a fund to make partial cash
payments to farmers upon delivery of produce. Once the groupmakes a sale the fund is replenished and farmers are given theremaining balance. “Cash on Delivery” (CoD)
• Assess the impact of this credit on the proportion of producesold through the group and on the price farmers received.
• Understand why this worked?• Did this work for farmers likely to face liquidity constraints, or
only in groups where trust was already high?• Implement an intervention on improved transparency to
randomly selected groups to improve trust in some groups. Isthe working capital intervention just as effective in thosegroups with the transparency intervention?
• Information on Sales (IoS): SMS system to provide memberswith specific information about transactions made by thegroup (final sale price, fees deducted, etc.), plus reinforcedtraining on book-keeping.
Impact evaluation An application to farmers groups in Uganda
Coffee/maize group marketing structure
• Farmers groups (“PO”s for producer organizations) aretypically grouped under associations (DCs for districtcommittees).
• The PO handles bulking and coordination of transport withmembers at the village level.
• The DC take care of collection and in some cases valueaddition to the next stage of marketing.
• In most cases, a service organization offers support to DCsand POs through lobbying, access to extension and additionalmarketing services.
Impact evaluation An application to farmers groups in Uganda
Implementation
• The study was carried out in 9 DCs marketing coffee andmaize, containing 165 POs under them.
• March 2010, Baseline survey:• A 3-tiered survey which collected detailed information on DCs,
POs, and member households.• Full roster of members for each PO, and a complete household
survey for at least 2 members of each group.
• November 2010 September 2011, implemented interventionin randomly selected groups:
• Provided working capital credit to randomly selected POs.• Provided SMS information on deliveries to randomly selected
POs.
• October 2011, Follow-up survey.• Collected detailed information on POs and member households.• Collected administrative data from the DC records to obtain
more reliable delivery data.
Impact evaluation An application to farmers groups in Uganda
Randomization strategy
• We randomized the interventions at the PO level stratifyingthe sample by DC, since the sample size is not large enoughat the DC level and the risk of spill-overs is too high at thehousehold level.
• POs in each DC are randomly assigned into 4 groups: (1)CoD, (2) IoS, (3) CoD + IoS, and (4) none.
• The fund for the CoD was managed by the DC, and voucherswere given to treated POs so their members could requestimmediate partial payments for output deliveries.
• For the IoS intervention, a DC staff member was selected tosend the messages to key farmers in the treated POs.
Impact evaluation An application to farmers groups in Uganda
Are control and treatment groups equal?
Control CoD IoS Both(mean)
Members 24.256 2.194 4.558 1.597(3.106)∗∗∗ (4.365) (4.289) (4.339)
Years since foundation 4.400 1.039 0.460 1.014(0.526)∗∗∗ (0.739) (0.731) (0.739)
Marketing services 0.825 -0.020 -0.081 -0.093(0.066)∗∗∗ (0.093) (0.092) (0.093)
Output bulked (kgs.) 854.025 -240.708 -192.862 -325.440(236.818)∗∗∗ (332.863) (329.018) (332.863)
Female leader 0.250 -0.006 -0.064 -0.030(0.067)∗∗∗ (0.094) (0.093) (0.094)
Leader’s age 52.200 -4.639 -2.153 0.190(1.874)∗∗∗ (2.634)∗ (2.604) (2.634)
Leader’s schooling 8.025 0.073 0.208 -0.562(0.460)∗∗∗ (0.647) (0.639) (0.647)
POs 40 41 43 41
Impact evaluation An application to farmers groups in Uganda
• Implementing the interventions represented a major challenge:
1. The POs in our study are spread over many regions in thecountry.
2. Implementation needed to be done by a 3rd party, to avoidservice organizations and DCs contaminating the PO-levelrandomization strategy.
3. In order to avoid undesired heterogeneity in implementation,training, and monitoring of the interventions, a singleimplementing agency was favored over several regionalorganizations.
4. Training and distribution of vouchers within the PO wasdelegated to PO leaders in some DCs.
• Our own monitoring activities as well as the follow-up surveyindicate implementation was problematic.
• Some cross-over and no-shows for CoD intervention• Overall implementation of IoS intervention.
Impact evaluation An application to farmers groups in Uganda
Empirical strategy
McKenzie (2011) shows that using baseline data on the outcomevariable of interest, allows more power to detect impact.Therefore, for our analysis we estimate:
Yi ,1 = α +∑j
γjDi ,j + θYi ,0 + εi ,1
Impact evaluation An application to farmers groups in Uganda
Results
Table 2.1: Impact of interventions on producedeliveries
PO Household
Kgs. P(Delivery) Kgs.
CoD only 747.826 0.186 162.700(325.294)∗∗ (0.079)∗∗ (88.400)∗
IoS only 355.764 0.089 62.660(320.386) (0.078) (87.060)
Both -584.566 0.101 122.000(455.073) (0.077) (86.280)
Observations 165 244 243R2 0.422 0.269 0.084
Impact evaluation An application to farmers groups in Uganda
Results
Table 2.2: Impact of selling through POon transaction features
Price Daysbetweensale andpayment
Sold through PO 0.858 -6.540(instrumented) (0.477)∗ (21.170)
Observations 193 192R2 0.704 0.210
Impact evaluation An application to farmers groups in Uganda
Concluding remarks
• Despite implementation problems, the CoD intervention has asignificant impact on group marketing.
• CoD increases the probability a household will sell through thegroup, how much each household will sell, and the totalamount sold by the group.
• By encouraging farmers to sell through the group, CoD has aneffect on increasing the price they receive.
•