developing a proxy means test for targeting welfare in sri
TRANSCRIPT
1
A Proxy Means Test for Sri Lanka
Ashwini Sebastian Shivapragasam Shivakumaran
Ani Rudra Silwal David Newhouse Thomas Walker1 Nobuo Yoshida
June 2018
Abstract This paper intends to inform the effort of the Sri Lankan government to reform the targeting efficacy of its social protection programs, in particular, Samurdhi, which currently distributes benefits based on self-reported income. We develop a Proxy Means Test (PMT) for Sri Lanka based on the Household Income and Expenditure Survey 2016 and evaluate its performance for targeting benefits of Samurdhi. The paper considers a range of models and policy parameters that could be applied depending on data availability and country preferences. The results indicate that switching to a PMT could considerably improve the targeting performance of Samurdhi and would significantly improve the poverty impact of the program. We find that the performance of the proposed PMT model suffers when the coefficients are estimated from samples smaller than 1,000 households. However, we do not find a similar loss of model performance when the model is estimated from seasonal data, provided the sample size is sufficiently large. The model we propose could be applied to the targeting of a variety of safety net programs after validating and refining our model by conducting a pilot survey. Keywords: Proxy means test; poverty; welfare; targeting
1 Corresponding author, email: [email protected]. All authors are affiliated with The World Bank. The authors are grateful to officials from the Sri Lanka Ministry of Finance, National Planning Department, Ministry of Social Empowerment and Welfare, Department of Census and Statistics, Ambar Narayan, Phillippe Leite, and Emil Tesliuc for their feedback and suggestions on an earlier drafts. All remaining errors and omissions are the responsibility of the authors.
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
2
I. Introduction Improved targeting has the potential to significantly increase the poverty impact of welfare spending in Sri Lanka. In 2002, 42 percent of all transfers provided by Samurdhi (the countryâs main social safety net program) were received by the poorest 20 percent of households.2 By 2012, this had slipped to 39 percent (World Bank, 2016c). Recognizing this, the government is currently conducting a review of the rules by which households are selected into welfare programs. The results of this paper are intended to inform that review. A proxy means test (PMT) is an index of observable and verifiable household characteristics that serves as a proxy for household welfare. The PMT is commonly used to target social safety net programs in situations where verifiable income data are not available. Since the PMT essentially ranks households by welfare level, the same formula can be used for targeting a range of different welfare programs, adopting different eligibility cutoff scores for each program and possibly adding other criteria as well.3 In this way, the PMT can serve as the basis for targeting of a range of different welfare programs. Having a common targeting approach for various programs, based on a single PMT score, ensures consistency of targeting across programs, minimizes overlaps, and provides a basis for future harmonization of programs. An earlier study by Narayan and Yoshida (2005) developed a PMT formula for Sri Lanka based on a household survey collected during 1999-20004. Sri Lanka has developed rapidly since then, however, and as the country has transitioned to middle income status, the poverty rate per the (official) national poverty line has fallen from 22.7 percent in 2002 to 4.1 percent in 2016. Due to these major demographic and structural changes, and to incorporate areas of the country not covered in the earlier work, this paper presents a PMT formula based on data from the 2016 round of the Household Income and Expenditure Survey (HIES). We benchmark the PMT to the targeting performance of the PMT for the Samurdhi program, which currently determines eligibility solely based on self-reported income. We find that switching to a PMT could significantly improve the targeting performance of Samurdhi. This paper examines only the PMT method, although we recognize that this is one of many targeting methods used around the world (Alatas et al., 2012; Alkire and Seth, 2013; Klasen and Lange, 2015; Brown et al., 2016; Diamond et al., 2016; Sabates-Wheeler et al. 2015; Del Ninno and Mills, 2015; Kydd and Wylde, 2011). This is because our goal is to inform the efforts of the Sri Lankan government, which already distributes Samurdhi transfers based on self-reported income and is in the process of updating this system, a component of which is to replace self-reported income with a vector of household attributes to identify beneficiaries (World Bank, 2016d).
2 The Samurdhi program provides monthly cash benefits to almost 1.5 million families, and accounts for over 80% of Sri Lankaâs cash transfer spending. The program also provides other assistance to its beneficiaries, including microfinance, housing, and banking services. 3 For instance, the disability benefit would need to include criteria relating to severity of disability. 4 Although the survey was nationally representative, data for the Northern and Eastern regions, affected by conflict at the time, were excluded for the purposes of the paper due to concerns about data quality.
3
In the next section, we explain the rationale for the PMT as a tool for measuring poverty. In Section III we describe the data and general set-up of the PMT formula. In Section IV we present results and discuss the selection of the best-performing models. This section demonstrates that PMT simulations outperform the targeting approach used in the Samurdhi program. Section V presents results of the robustness checks that we conduct on our primary PMT model. The final section presents conclusions and suggested next steps. final section presents conclusions and suggested next steps. II. The Rationale for a Proxy Means Test PMT-based targeting has been adopted in many low- and middle-income countries since the 1990s. Case studies on performance in terms of targeting incidence suggest that the PMT approach has been very successful and outperformed other targeting mechanisms (Grosh, 1994). For example, in Chile and Mexico approximately 90 percent of social assistance reached the bottom 40 percent of the population when a PMT was adopted (Castañeda and Lindert, 2005). Beyond Latin America, Armenia introduced a PMT in 1994 for targeting cash transfers (World Bank, 1999, 2003), Indonesia has adopted PMT for targeting subsidized rice rations and transfers following the removal of fuel subsidies (Sumarto et al., 2000), and Turkey introduced PMT-based targeting in 2002 (Ayala, 2003). In the South Asia region, PMT-based targeting systems have been employed in Pakistan and Bangladesh (Sharif, 2009; Hou, 2008). The PMT determines eligibility for a program based on an index of easily observable household characteristics, such as the quality of the householdâs dwelling, ownership of durable goods, demographic structure, and the education and employment status of adult household members (Grosh et al., 2008). Statistical analysis of data from household surveys is normally used to refine the set of indicators included in the index and to estimate their respective weights. Information collected from each program applicant is then used to construct a household-specific âPMT scoreâ, which is a proxy for the familyâs relative welfare level in the population. This score is used to determine eligibility for safety net programs and potentially also the level of benefits. A household is eligible for the program if its PMT score falls below a predetermined cutoff score. Although actual household income is in theory a more accurate measure of welfare (and indeed self-reported income is currently used as a main eligibility criterion for the Samurdhi program), it can be difficult to accurately measure or estimate reliably in countries like Sri Lanka, where a large proportion of households are self-employed or informally employed. 5 The value of informal transfers like gifts and remittances is also difficult to measure. If income self-declarations are difficult to verify, households have a temptation to understate income in order to qualify for benefits. In cases like Sri Lanka where accurate data on household consumption is also burdensome to collect and difficult to verify, self-reported consumption faces similar drawbacks as a targeting approach. The PMT offers a way to estimate a householdâs welfare level without utilizing self-declared income or consumption.
5In the first quarter of 2016, one third of all workers were own account workers (Department of Census and Statistics, 2016).
4
Because the PMT formula provides a prediction of consumption, it is susceptible to inaccuracies, both by including ineligible households and excluding eligible households. Figure 1 depicts the two different types of targeting errors that arise. In this example, the target group consists of all households below the poverty line. âInclusion errorsâ are cases in which households who are above the true cutoff score (the poverty line) get accepted into the program. âExclusion errorsâ are cases where poor households fall above the cutoff score and are refused entry into the program. The magnitude of these âinclusionâ and âexclusionâ errors depends on the level of poverty and the cutoff score. For example, as the poverty rate in a country increases, holding the cutoff constant, more of the poor will be excluded from selection at the same time that fewer non-poor are included. Similarly, reducing the cutoff score increases exclusion errors, while reducing inclusion errors. Balancing these errors is a key component of selecting cut-offs using a PMT. Recent empirical evidence from PMT targeting in Pakistan and Bangladesh shows that targeting less than about 20 percent of the population leads to many poor households being excluded, but also reduces the number of non-poor households that are included (Hou, 2008; Sharif, 2009). Minimizing such errors also requires care in data collection to ensure accurate assessment of household poverty (Grosh & Baker, 1995; Sharif, 2009). Including other information (such as verifiable income) in the beneficiary selection process can also help reduce targeting errors. Figure 1. Interpreting the targeting performance of programs*
* Note: The target population is all households below the poverty line. III. Methodology We determine the set of variables to be used in the final PMT formula based primarily on statistical analysis of household survey data. The goal is to identify a small number of objective and verifiable household characteristics that together provide a good proxy for a householdâs welfare. Each irrelevant characteristic
5
decreases the precision of the model and increases the complexity and cost of data collection. Candidate variables should be well correlated with poverty and have the following three characteristics (Coady et al., 2004):
âą The set of variables should be small enough that it is feasible and cost-effective to collect the required data from a significant share of the population.
âą Each selected variable must be easy to measure and observe. âą Each selected variable should be difficult for the household to manipulate.
A broad range of candidate variables is typically drawn from a detailed survey datasetâfor example, a household budget survey or a multi-topic surveyâthat includes detailed information on consumption, employment, education, health, housing, and family structure for a population representative sample. In this paper, we use HIES 2016, the most recent detailed household survey of this type available. We first reduce the set of candidate variables through analysis of variance (ANOVA), in which we try to model the determinants of the log of per capita household consumption. We use ANOVA to discard variables with low partial sum of squares (SS), in other words, variables that are poor at explaining variation in consumption. This yields three plausible models of the determinants of the log of per capita household consumption. The first is a typical PMT model that includes characteristics of the household head but does not include any variables with a SS below 2. The second is restricted to variables with SS above 3.5, and excludes characteristics of the household head. The third model is much more selective and drops any variables with a SS below 10.6 This exercise was initially conducted using the HIES 2012/13 and feedback was obtained from various Sri Lankan government agencies during 2015-17 before finalizing the model specifications. An OLS regression of the log of per capita household consumption on the three model specifications is then conducted to simplify interpretation of the magnitude and significance of coefficients for predicting poverty. We then measure the extent to which the model correctly identifies the poor. Following Narayan and Yoshida (2005), individuals are categorized into four groups according to whether their true and predicted welfare levels fall above or below the defined cutoff score. Those whose true welfare falls below the cutoff score constitute the target group, while those with predicted welfare below the cutoff score constitute the eligible group. Individuals whose true and predicted welfare measures put them on the same side of the cutoff are considered targeting successes. Prediction errors in the model may lead some eligible households to be excluded from the program (exclusion errors) while other ineligible households are included (inclusion errors). These are also known respectively as Type I and Type II errors, and are summarized in Figure 2. The undercoverage rate is calculated by dividing the number of cases of Type I error by the total number of individuals who should get benefits (e1/n1). Increased undercoverage reduces the impact of the program on the welfare level of the intended beneficiaries, but carries no budgetary cost. The leakage rate, on the other hand, is calculated
6 In addition to some characteristics of head of household, the only exception is the variable âdrinking water source insideâ, which does not satisfy the SS cutoff criteria but is left in the model because it is considered to be an important predictor of poverty.
6
by dividing the number in the Type II error category by the number of persons served by the program (e2/m1). Leakage has no effect on the welfare impact of the program on the intended beneficiaries, but increases program costs. We define the PMT eligible share as the share of the total population eligible for benefits at a given threshold (m1/n). This includes the correctly and incorrectly identified eligible population (using the PMT formula), as a share of the total population.
Figure 2. Illustration of Type I and Type II errors
While it is desirable to reduce both undercoverage and leakage, there is a tradeoff between the two. The ultimate choice of cutoff needs take into account the size of the budget and the average benefit per recipient household. In analyzing the Samurdhi program, we present a range for this tradeoff while varying the eligibility cutoff. We also evaluate the incidence of targeting based on the PMT model, i.e. the extent to which the model selects beneficiaries with per capita consumption towards the bottom of the consumption distribution, and selects relatively few, if any, from the wealthier households at the top of the distribution. Once a beneficiary group is selected, however, it is important for programs to implement a mechanism to review cases of potential exclusion errors. We recommend designing a grievance or appeals process to review particular households that may have been excluded due to scoring higher on one or two important predictors. For example, smaller households are less likely to be poor and therefore more likely to receive a higher PMT score. However, this may miss smaller vulnerable households like those headed by elderly people caring for orphaned grandchildren. A grievance process can help address specific cases of exclusion like these. Data and Variables The HIES is a multi-topic survey conducted every three years by Sri Lankaâs Department of Census and Statistics (DCS). The survey includes modules on consumption, education, health, income, household characteristics, assets, and information on access to social safety net programs. The nationally representative survey included 21,756 households from all regions of Sri Lanka, including areas of the North and East that were excluded from previous rounds due to conflict. In this study, we aim to develop a PMT formula that is consistent across different definitions of a household. The HIES includes data collected at the household level, while eligibility of Samurdhi is determined at the
Target group Non-target group Total
Eligible: predicted by PMT formula
Targeting Success (s1) Type II error (e2) m1
Ineligible: predicted by PMT formula
Type I error (e1) Targeting Success (s2) m2
Total n1 n2 n
7
family level. Households are defined as those in a dwelling that share a cooking pot, while families are defined according to nuclear family relationships.7 It is therefore possible that more than one family may reside in a single household. Partly because of this, we present three models of the PMT: Model 1 includes 36 predictors of consumption, inclusive of 9 household head variables. Model 2 excludes head characteristics, and therefore includes 27 predictors, while a more parsimonious Model 3 has only 18 predictors. 8 Models 2 and 3 exclude head characteristics because current targeting of the Samurdhi program is done at the family level, and there may be significant differences between household and family head information. Although including more variables predicts welfare slightly better, including more variables in the model makes data collection more costly and increases the chance of errors. PMT models used in other countries are usually similar to Model 1 and include characteristics of the household head. In fact, the results of the three models are not substantially different from each other, and all three lead to considerably better targeting than the existing selection method. While income typically fluctuates as a result of both permanent and transitory income shocks, households typically smooth consumption over time, so we use the log of per capita household consumption as the measure of welfare. Consumption comprises monthly expenditure on both durables and non-durables, as well as the reported rental value of owner-occupied housing. The DCS has published official poverty lines for each district based on the 2016 HIES. We use these district poverty lines to construct a district price index, and deflate the consumption measure by this index.9 This enables us to set a single cutoff score for households across the country. The PMT score for a household is created by multiplying its predicted log real per capita consumption by 100. The household head characteristics included in Model 1 are age, gender, marital status, and employment status (government, semi-government, private, or self-employment). Household demographics include categorical household size variables, the dependency ratio (the share of all household members that are older than 64 or younger than 15), and the highest education level of household members not currently enrolled in school. We include several household assets, including ownership of a computer, cooker, electric fan, refrigerator, land phone, washing machine, water pump, motorcycle, car or van, three-wheeler, and four-wheel tractor. Housing quality and facilities variables included are bedrooms per person, an inside drinking water source, the use of electricity for lighting, the presence of floor tiles/terrazzo, walls made of brick/cabok/cement, and the presence of an inside toilet. Due to the household/family distinction, instead of considering the number of rooms within a household, we use rooms per capita.10 Further, we consider the highest education level of all household members.
7 However, the precise definition of a family used by the Samurdhi program does not seem to exist, possibly to allow for context-specific interpretations of a family. 8See Appendix Table A1 for a comparison across models used in other papers developing a PMT in South Asia including Narayan and Yoshida (2005), Sharif (2009) and Hou (2008). 9 The spatial price index for a district is computed as the ratio of district poverty line to national poverty line. Spatially price adjusted consumption is computed by deflating the nominal consumption by relevant spatial price indices of the districts. DCS computed the district poverty lines by using a spatial price index, considering the price variations among districts for the selected basket of food items consumed as per HIES 2016 survey. 10 We expect rooms per capita will be more similar for families and households than total rooms in the household.
8
Ownership of a four-wheel tractor is included in Models 1 and 2 to assess the importance of agricultural livelihood-related assets in determining poverty. However, this variable was dropped in Model 3 due to the potential for excluding poor agricultural households that own a tractor. We also drop land phone and water pump variables in Model 3. Land phones are likely to become obsolete over time, and just 2 percent of households own water pumps, limiting the contribution of these variables as predictors of poverty. Other variables (see Table A1) were dropped from Model 3 because of their limited explanatory power. We also tested he inclusion of district and sector categorical variables in the model, but this did not alter the results significantly.11 IV. Results After several iterations of models testing the inclusion of different variables, we present three models that fit the HIES data well (Table 1). Model 2 differs from Model 1 only in the exclusion of household head variables. Model 3 is a restricted version of Model 2, retaining only highly significant coefficients, with partial sum of squares above particular thresholds (see section II above). Model 1 is the typical PMT model. If the program eligibility continues to be determined at the family level, however, Model 3 would be preferable, because it performs almost as well as Model 2 and requires less intensive data collection and verification. All models generate high R-squared statistics, with Model 3 having a slightly lower R-squared as expected due to the lower number of variables. In Model 1, several of the household head characteristics are not significant contributors to the PMT score. However, widowhood of the head, age, and head working in a government, semi-government or self-employed job all affect the PMT score significantly. Across all models, categorical household size variables are the largest contributors to the PMT score, with the score decreasing as household size increases. This is typically the case in a PMT because welfare is defined in (log) per capita terms, which mechanically makes household size a crucial predictor. The remaining coefficient contributions to the PMT score are fairly straightforward, with asset ownership and housing quality contributing positively to the score.
11Because of their small impact, and the potential objections that could be raised from including such variables, they are omitted from the models presented here. Results from specifications including location variables are available from the authors upon request.
9
Table 1. Estimated PMT Models Model 1 Model 2 Model 3 Variables Mean Coeff. S.E. Coeff. S.E. Coeff. S.E. HH head characteristics
Female head 0.22 0.00 (0.01) Marital status of head (Reference category: never married)
Married 0.82 0.00 (0.03) Widowed 0.14 0.01 (0.03) Divorced 0.00 -0.01 (0.05) Separated 0.02 -0.04 (0.03)
Age 52.30 -0.00*** (0.00) Employment status of head (Reference category: Not employed, Own account worker, or Contributing family worker)
Govt/semi-govt employee 0.10 0.10*** (0.01) Private employee 0.31 0.00 (0.01) Employer 0.02 0.20*** (0.02)
HH demographics
Household size (Reference category: 6 or more members) 1 member 0.02 0.82*** (0.03) 0.83*** (0.03) 0.73*** (0.03) 2 members 0.08 0.58*** (0.02) 0.58*** (0.02) 0.54*** (0.02) 3 members 0.17 0.41*** (0.01) 0.42*** (0.01) 0.40*** (0.01) 4 members 0.28 0.27*** (0.01) 0.29*** (0.01) 0.28*** (0.01) 5 members 0.24 0.16*** (0.01) 0.17*** (0.01) 0.17*** (0.01)
Highest education level of members not currently enrolled (Reference category: less than primary or no education) Grade 10 0.43 0.09*** (0.02) 0.09*** (0.02) O/L 0.22 0.14*** (0.02) 0.14*** (0.02) A/L 0.24 0.22*** (0.02) 0.22*** (0.02) 0.12*** (0.01) University degree 0.07 0.35*** (0.02) 0.35*** (0.02) 0.24*** (0.01)
Dependency ratio 0.36 -0.12*** (0.01) -0.13*** (0.01) -0.14*** (0.01) HH assets (yes=1)
Computer 0.22 0.12*** (0.01) 0.13*** (0.01) 0.14*** (0.01) Cooker 0.56 0.19*** (0.01) 0.19*** (0.01) 0.20*** (0.01) Electric fan 0.64 0.12*** (0.01) 0.12*** (0.01) 0.13*** (0.01) Refrigerator 0.55 0.11*** (0.01) 0.11*** (0.01) 0.12*** (0.01) Washing machine 0.22 0.13*** (0.01) 0.14*** (0.01) 0.14*** (0.01) Land phone 0.30 0.05*** (0.01) 0.04*** (0.01) Water pump 0.02 0.16*** (0.02) 0.15*** (0.02) Motorcycle 0.40 0.10*** (0.01) 0.11*** (0.01) 0.12*** (0.01) Car/van 0.10 0.40*** (0.01) 0.42*** (0.01) 0.42*** (0.01) Three-wheeler 0.16 0.14*** (0.01) 0.14*** (0.01) 0.15*** (0.01) Four-wheel tractor 0.01 0.20*** (0.03) 0.20*** (0.03)
Housing quality and facilities
Bedrooms per person 0.64 0.15*** (0.01) 0.14*** (0.01) 0.17*** (0.01) Have floor tiles/terrazzo 0.18 0.14*** (0.01) 0.15*** (0.01) 0.16*** (0.01) Drinking water source: inside unit 0.61 0.03*** (0.01) 0.02*** (0.01) Electricity for lighting 0.98 0.09*** (0.02) 0.08*** (0.02) Have wall of brick/kabok/cement 0.93 0.02 (0.01) 0.02 (0.01) Have toilet within unit 0.98 0.07*** (0.02) 0.07*** (0.02)
Constant 8.34*** (0.05) 8.22*** (0.03) 8.49*** (0.01) Observations 21756 21756 21756 R-squared 0.533 0.526 0.520 Note: The score weights are derived by multiplying coefficient values by 100. A given householdâs PMT score is then the sum product of the weights with the householdâs variable values, including the constant (i.e. for Model 1, adding variable scores for household to constant of 834). The dependency ratio is defined as the share of members younger than 15 years and older than 65 years in the household.
10
The top panel of Table 2 displays the proportion of the population in different quantiles of the true per capita consumption quantiles (columns) deemed eligible for Samurdhi benefits by Model 1. The PMT cutoff scores (rows) are set at the 20th, 30th and 40th percentile of the true per capita consumption distribution, implying that around 20, 30, and 40 percent of the population with scores below the respective cutoffs would be eligible for benefits.12 As is evident from the table, the share of households eligible for benefits increases monotonically as the cutoff rises. According to the HIES, 40.2 and 37.1 percent of population in the bottom decile and quintile, respectively, were beneficiaries of Samurdhi in 2016. By applying a PMT with the weights estimated in Model 1, around 75 percent of the bottom decile, and 65 percent of the bottom quintile, would be eligible if the PMT cutoff score were established at the 30th percentile of the true consumption distribution. This rule would select around 24 percent of the population overall. Table 2. Comparative Targeting Performance of Model 1
Quantiles of the true consumption distribution=> Bottom Decile
Bottom Quintile
Top Quintile Top
All Decile
Eligible share Existing Samurdhi beneficiaries 40.2 37.1 3.8 2.2 18.8 PMT cutoff score*
Score: 856 (10th percentile) 17.5 12.0 0.0 0.0 2.9 Score: 868 (15th percentile) 34.3 25.0 0.1 0.0 6.8 Score: 878 (20th percentile) 49.3 39.1 0.3 0.1 11.9 Score: 887 (25th percentile) 64.3 53.6 0.5 0.3 17.5 Score: 895 (30th percentile) 75.8 65.5 0.9 0.5 23.7 Score: 902 (35th percentile) 84.7 75.6 1.9 1.1 30.2 Score: 909 (40th percentile) 90.0 83.0 3.0 1.8 36.6
Distribution of beneficiaries Existing Samurdhi beneficiaries 21.4 39.5 4.1 1.2 100 PMT cutoff score*
Score: 856 (10th percentile) 59.9 82.1 0.0 0.0 100 Score: 868 (15th percentile) 50.5 73.8 0.1 0.0 100 Score: 878 (20th percentile) 41.6 66.0 0.6 0.1 100 Score: 887 (25th percentile) 36.7 61.1 0.6 0.2 100 Score: 895 (30th percentile) 32.0 55.2 0.8 0.2 100 Score: 902 (35th percentile) 28.1 50.1 1.2 0.4 100 Score: 909 (40th percentile) 24.6 45.4 1.7 0.5 100
Notes: Similar PMT eligible share and distribution of beneficiaries statistics for Models 2 and 3 can be found in Appendix table A2. * The PMT score cutoffs represent different percentiles of the true per capita consumption distribution. For example, the score 878 is derived by multiplying the 20th percentile of the log of the true consumption distribution by 100. The bottom panel of Table 2 shows the distribution of beneficiaries by top and bottom consumption deciles and quintiles for various cutoff scores under Model 1. We find that at all cutoffs, more of the program benefits would accrue to the poor than at present. If Samurdhi was targeted based on a PMT, more than 50 percent of program beneficiaries would fall come from the poorest quintile at the 20 and 30 percent cutoffs, compared to 39.5 percent based on the programâs current beneficiary population. In addition, a much smaller share of the program would go to the top quintile or decile if beneficiaries were selected by PMT than the current methodology. The targeting performance is considerably better than the existing
12 Scores set at the 20th ,30th and 40th percentiles are 852, 868 and 882 respectively.
11
Samurdhi allocation rule, increasing the poorest quintileâs share of benefits from 39.5 percent to 55.2 percent (at the 30% cutoff). Another important goal is to minimize selection of richer households. Table 2 shows that the PMT far outperforms existing Samurdhi targeting in this respect as well. At the 30 percent cutoff, 0.8 and 0.2 percent of households in the top quintile and decile would be eligible for benefits, compared to an estimated 4.1 and 1.2 percent of those in the top quintile and decile that currently benefit from Samurdhi. The inclusion error, or the leakage rate, also increases as the cutoff score increases (Table 3). In setting the cutoff, therefore, policymakers face a trade-off between undercoverage and leakage given a fixed program size and budget. Table 3. Undercoverage, leakage and eligible share by cutoff score and model for the target population of Bottom 25%
Model 1 Model 2 Model 3
Cutoff score Under-coverage
Leakage
Eligible share
Under-coverage
Leakage
Eligible share
Under-coverage
Leakage
Eligible share
Score: 856 (10th percentile) 89.6 11.5 2.9 89.9 12.4 2.9 90.8 11.8 2.6
Score: 868 (15th percentile) 78.1 19.5 6.8 77.9 17.9 6.7 78.2 19.8 6.8
Score: 878 (20th percentile) 64.6 25.5 11.9 65.1 24.7 11.6 66.7 25.2 11.1
Score: 887 (25th percentile) 51.3 30.5 17.5 52.5 30.6 17.1 52.3 31.0 17.3
Score: 895 (30th percentile) 39.3 36.0 23.7 40.0 36.0 23.5 41.0 36.6 23.2
Score: 902 (35th percentile) 29.2 41.3 30.2 30.1 41.1 29.7 30.6 41.7 29.8
Score: 909 (40th percentile) 21.0 46.1 36.6 21.8 45.9 36.1 22.0 46.4 36.4 * The PMT score cutoffs represent different percentiles of the true per capita consumption distribution. For example, the score 887 is derived by multiplying the 25th percentile of the log of the true consumption distribution by 100. The results from Table 2 and Table A2 collectively present a clear argument for the consideration of the proposed PMT model for targeting Samurdhi. A significantly higher share of the poor can be reached, and inclusion errors from the program can be simultaneously minimized relative to the programâs current targeting method. Models 2 and 3 perform a few percentage points worse than Model 1 in undercoverage and leakage at all cutoffs (Table 3). The advantage of Model 3 is its use of fewer predictors, making the model simpler and cheaper to implement, but at the cost of slightly higher undercoverage and leakage. Choice of Target Population Sri Lankaâs official poverty headcount, based on the HIES 2016 data, was 4.1 percent. That survey reveals 18.8 percent of the population to be Samurdhi beneficiaries, but the true proportion is believed to be 20-25 percent due to underreporting among survey respondents. For this reason, we recommend setting the target population for Samurdhi as the bottom 25 percent. Table 4 presents the shares of beneficiaries identified by PMT, by (actual) consumption deciles under different cutoff rates and PMT models. The eligibility rates for all three models are more or less similar, with overall coverage of 17.5, 23.7 and 36.6 percent for target populations of 25, 30, and 40 percent
12
respectively. In order to target the bottom 25 percent, it is recommended to set the cutoff score at the 30th percentile (which would make 23.7 percent of households eligible). The corresponding undercoverage rate for the three models is between 39 and 41 percent, and the leakage rate is about 36 percent. Using Model 1 with a 30 percent cutoff, the leakage rate would be 36 percent with 23.7 percent of the population eligible for Samurdhi. More than 75 percent of the bottom decile would be eligible for Samurdhi (compared with around 40 percent currently; see Table 2), and less than 1 percent of households in the top quintile would be selected (versus 3.8 percent at present). Table 4. Eligibility rates for deciles of the actual consumption under different PMT cutoff scores and PMT models
Decile
Model 1 Model 2 Model 3
Score: 887
(25th percentile)
Score: 895 (30th
percentile)
Score: 909 (40th
percentile)
Score: 887 (25th
percentile)
Score: 895 (30th
percentile)
Score: 909 (40th
percentile)
Score: 887 (25th
percentile)
Score: 895 (30th
percentile)
Score: 909 (40th
percentile)
1 64.3 75.8 90.0 62.7 75.2 89.3 62.5 74.5 89.2 2 42.8 55.1 76.0 41.5 54.8 75.1 42.0 53.4 74.8 3 27.0 38.3 59.9 26.7 36.9 58.6 27.2 36.7 58.2 4 17.2 27.4 48.8 16.9 26.9 47.7 17.5 26.7 48.4 5 9.8 17.0 34.5 9.9 16.9 33.6 9.7 16.4 34.0 6 7.6 12.3 25.8 7.7 12.5 25.2 7.4 12.4 25.6 7 3.5 5.8 15.8 3.2 5.9 15.6 3.7 6.7 16.4 8 2.0 3.6 9.2 1.7 3.5 9.7 1.9 3.8 10.3 9 0.8 1.4 4.2 0.7 1.5 4.8 0.8 1.4 5.2
10 0.3 0.5 1.8 0.2 0.5 1.6 0.2 0.4 1.8 Total 17.5 23.7 36.6 17.1 23.5 36.1 17.3 23.2 36.4
Welfare impact of improving targeting efficiency We have so far examined the PMT formula from the perspective of targeting efficiency. A related, and equally important question is: what kind of welfare improvements we can expect with the application of these formulas in transferring benefits to eligible beneficiaries? We present one case which could serve as a useful example, although there are multiple ways in which we could answer this question. Narayan and Yoshida (2005) discuss more scenarios and describe the basic methodology in detail. Table 5 presents the welfare outcomes, measured by the Foster-Greer-Thorbecke (FGT) index under three scenarios. The first case is the baseline data collected in HIES 2016, in which household consumption includes Samurdhi benefits. The poverty head count at the national poverty line of Rs. 4,166 per month and the $3.20 per day for lower middle-income countries in this baseline scenario are 4.08% and 9.50%, respectively. In the second scenario, Samurdhi benefits are subtracted from the consumption expenditures of all households. The third scenario modifies the second scenario by re-introducing the Samurdhi benefits (Rs. 1732 per capita) for households in the bottom 20% of the consumption distribution, identified by the PMT formula and a 25th percentile cutoff score. The amount Rs. 1732 was obtained by dividing the total Samurdhi benefits received by all households, according to HIES 2016, by the population in the bottom 20% of the welfare distribution. This new allocation
13
rule would effectively raise Samurdhi benefits by five times for eligible households, but assume that the benefits will be perfectly targeted towards the poorest 20% of the population.13 Bottom 20% is chosen as the target population since about 20% of the population currently receives Samurdhi benefits. The PMT score cutoff of 25% is used to identify approximately the bottom 20% households based on Table 3. Scenario 3 in Table 5 is budget neutral compared to Scenario 1 since the total Samurdhi benefits subtracted in Scenario 2 are simply re-distributed among eligible households according to their PMT score. The consumption values and Samurdhi payments in all of the scenarios are conducted at the per capita level in order to make the scenarios comparable. Table 5: Comparison between different transfer schemes
Scenarios Poverty
Headcount index Poverty
Gap Index Poverty
Severity Index α =0 α =1 α =2 1. Baseline scenario (Per capita consumption)
a. National Poverty Line 4.08% 0.006 0.002 b. $3.20 International Poverty Line 9.73% 0.018 0.005 c. $5.50 International Poverty Line 39.67% 0.113 0.044
2. Per capita consumption minus Samurdhi payment
a. National Poverty Line 4.87% 0.009 0.003 b. $3.20 International Poverty Line 10.92% 0.022 0.007 c. $5.50 International Poverty Line 40.58% 0.121 0.050
3. Household consumption minus Samurdhi payment plus Rs.1732 (all per capita) for eligible households identified by the PMT formula
a. National Poverty Line 1.63% 0.003 0.001 b. $3.20 International Poverty Line 5.50% 0.009 0.002 c. $5.50 International Poverty Line 37.01% 0.087 0.030
Note: The poverty line for this exercise is set at Rs. 4,166 per person per month, the official national poverty line in 2016. The international poverty line is in 2011 PPP dollars per person per day converted to 2016 Rupees. The α refers to the parameter in the FGT index. Analysis based on HIES 2016. Scenario 3 allows for some amount of mistargeting since the redistribution is based on the PMT formula. The $3.20 and the $5.50 lines refer to the thresholds for Lower Middle-Income Countries and Upper Middle-Income Countries, respectively, according to World Bank definitions. The last three columns of Table 5 present the following FGT measures for the scenarios we examine: poverty headcount index (α=0), poverty gap index (α=1), and poverty severity index (α=2). We see that the poverty headcount rate set at the national poverty line and the $3.20 line are 4.08 percent and 9.73 percent, respectively, in the baseline scenario. In this scenario, households report receiving Samurdhi benefits in the HIES 2016. In the second scenario, we remove the Samurdhi payments that households may have been receiving. As a result, we see that the poverty headcount rate increases to 4.87 percent and 10.92 percent in this scenario due to lower consumption levels. In the third scenario, in which recipient households are identified using the PMT formula (Model 1) presented in this paper is given Rs. 1732 per
13 Using the governmentâs actual budget on Samurdhi payments during 2016 would be more realistic than the scenario examined here.
14
person per month. The resulting increase in household consumption lowers the national poverty measures. We see that removing the Samurdhi payments from households would increase the poverty headcount index by 0.79 and 1.19 percentage points, respectively, under the national and $3.20 poverty lines. However, results from the third scenario suggest that the poverty would drop to 1.63 percent and 5.50 percent, under the national and $3.20 lines, respectively, if these budget-neutral transfers were made to beneficiary households identified by the PMT formula. These results suggest that using the PMT method would significantly improve the poverty impact of Samurdhi transfers by effectively targeting benefits towards the most needy households. V. Robustness This section examines the robustness of our PMT model to various issues. First, we examine performance of the PMT model when the analysis is conducted with different sample sizes. Second, we examine how much the performance of the model varies by whether individual survey quarters or the full annual sample is used to estimate the PMT coefficients. We finally examine PMT performance at sub-national levels and for certain demographic groups. PMT performance and sample size We often rely on household surveys to design, verify, and refine the PMT model. Cost and logistical considerations are often an important factor when implementing household surveys. This raises the following question: how much does changing the sample size of the household survey affect the performance of the PMT model built from it? Having the answers to this question would help relevant organizations optimize scarce resources. We address this question by taking random subsamples of HIES 2016, whose surveyed 21,765 households. We do not change variables in the PMT model presented in Table 1, although we recognize that the variables in the PMT model would depend on the specific sample used to design it. We estimate the PMT coefficients, assign PMT scores to households, and simulate the beneficiaries based on the cutoffs also computed from samples of varying sizes. We then examine the performance of the PMT model in terms of undercoverage, leakage, eligibility, and estimated poverty rates for varying rates of sample sizes.
15
Figure 3: PMT performance and sample size
Figure 3 presents the results of these simulations. Each tick mark on the X-axis represents results from a simulation of a certain sample size. The different lines represent undercoverage, leakage, eligibility, and poverty rates estimated for different samples. Figure A1 plots PMT scores estimated from the full sample on the Y-axis against the one estimated from smaller sample on the X-axis. In Figure 3, we see that undercoverage is the lowest for the smallest samples. This is because the consumption distributions predicted by the PMT model based on larger samples have thinner tails than the distribution of actual consumption based on the full sample (Figure A2).14 In other words, consumption estimated from larger samples assigns fewer people as poor than there actually exist at any given poverty line. We can also see this in the yellow line in Figure 3, which is sloped slightly downwards. The graph also suggest a trade-off between undercoverage on the one hand and leakage and eligibility on the other hand for samples smaller than 1,000 households. We see the lines leveling off between sample size of 750-1,000 households, suggesting the to be the optimal sample size to be around 1,000 households. PMT performance and survey period Another important question in the design of PMT pilots is the survey period. Ideally, the survey would be fielded uniformly across the quarters so that the consumption estimates (which often rely on a recall or diary of consumption in the previous week or two weeks) are not biased by seasonality. However, doing this in practice may not be feasible due to logistical or budgetary reasons. This raises the following question: how does the performance of the PMT model change across different calendar quarters? We address this question by implementing Model 1 on quarterly subsamples of HIES 2016. In other words, we do not change the variables in the PMT model, but only estimate its coefficients for the quarterly subsamples and predict consumption that can be rescaled to obtain the PMT scores. These PMT scores and the actual consumption values are then used to measure the performance metrics of undercoverage, leakage, and eligibility. 14 The Law of Large Numbers suggests that the estimate of the consumption based on larger samples will have a smaller variance than estimates based on smaller samples.
0
10
20
30
40
50
60
Undercoverage (%) Leakage (%) Eligibility (%) Poverty rate (%)
Perc
ent
N=100 N=250 N=500 N=750 N=1000 N=10000 Full sample (N=21,757)
16
Figure 4 compares the performance of PMT Model estimated using the (full) annual and the quarterly samples. We see some variation in the undercoverage, leakage, and eligibility across the quarters, although this variation is not nearly as large as we see for different sample sizes in Figure 3.15 This result alleviates some concern that a PMT pilot that is only conducted during a few months of a year may still give valid results provided it is not based on a very small sample. Figure 4: PMT performance and survey quarter
Note: HIES 2016 was conducted during January-December 2016. The surveys were conducted roughly evenly across the survey months. The variables in the PMT model are the same for all the samples but PMT coefficients are estimated for each sample.
Although HIES interviews were conducted fairly uniformly across the survey quarters, we find some differences in the sample sizes across the quarters. Could the differences in PMT performance we observe across quarters due to sample size in addition to seasonality? Figures A3-A5 presents results of the simulations in which we draw random samples of 4500, 1000, and 250 households, respectively, in each quarter.16 We also present three scenarios for the annual sample: same sample size drawn from the full year, sum of all the quarterly samples, and the all 21,756 households interviewed in the survey. We see that the performance of the PMT models varies slightly more when we use constant (and smaller) samples compared to Figure 4 in which the full available sample was used across the quarters. The model performance varies significantly when we use very small samples of 250 households per quarter. The takeaway from Figures 4 and A3-A5 is that although variation in PMT performance across quarters may not be a concern if large samples are used. PMT performance at sub-national levels
15 The differences in poverty rate across the quarters may simply reflect the seasonality of household consumption, which is why a national estimate of poverty should be based on a survey that covers all months. 16 The sample size of 4,500 was chosen to maximize the number of observations per quarter while keeping it constant across the quarters (Q4 surveyed only 4,511 households). The sample size of 1,000 was chosen since we previously found that this may be the sample size beyond which gains in the performance of the PMT model may be very small. The sample size of 250 was chosen to allow the full sample to be 1000 households.
0
10
20
30
40
50
60
Undercoverage (%) Leakage (%) Eligibility (%) Poverty rate (%)
Q1 (N=5,429) Q2 (N=5,540) Q3 (N=5,056) Q4 (N=4,511) Full/annual sample
17
How does the performance of the PMT differ across provinces and districts? In Table A3 we present undercoverage, leakage and PMT eligible share by province and district, using a 30 percent cutoff score and taking the bottom 25 percent as the target population (i.e. the poor and near-poor). Separating leakage and undercoverage rates by district is important to determine whether the model causes disproportionate error rates for any particular areas. As expected, there is considerable variation in the share of population in each district that would be eligible, and therefore in the undercoverage and leakage rates. For example, using the PMT formula, only 9.8 and 11.7 percent of households in Colombo and the Western Province respectively would be eligible, reflecting its low incidence of poverty. Undercoverage is highest in the Western Province, at 49-52 percent. This is because the poverty rate is low in that province, and the small share of poor people is more difficult to target. On the other hand, in Northern province, where poverty rates is high, we find higher coverage and very low leakage rates. PMT performance for different demographic groups Tables A4 and A5 provide sensitivity analysis by breaking down targeting performance by sector, household size and age of the household head. In Table A4 we find the highest eligibility in the estate sector, which tends to be disadvantaged in a variety of ways, and the lowest eligibility in the urban sector. These results are similar to the results reported in Narayan and Yoshida (2005). In Table A6 it is evident that the share of households in the population and the bottom 25 percent is larger both as household size and the age of the household head increase. However, the current proportion of Samurdhi beneficiaries does not increase with household size. Using the PMT formula, eligibility rises in line with increases in the share of poor as household size increases. Similar results can be found when considering the age of the household head. It would be advisable to first validate the PMT model in the field before applying it in practice, to determine whether adjustments to the selection criteria are needed. For example, while smaller households and those with heads under the age of thirty are less likely to be poor on average, certain portions of these groups may be more in need of access to social programs. This necessity may not be captured through just the PMT formula, which evaluates eligibility based on a given set of criteria applied to average statistical relationships for the entire population. Additional selection criteria may also be region specific. Such adjustments were found necessary during the pilot of the model developed by Narayan and Yoshida (2005). Other robustness checks: Sectoral models, quantile regression, estimators, and out-of-sample test This section presents results from a few additional robustness exercises of the PMT model presented in this paper. The first exercise examines if targeting performance improves if PMT scores are generated using separate PMT models for urban, rural, and estate sectors instead of a single model for the entire country. Table A6 presents coefficients of the national model along with individual models for urban, rural, and estates. There are some large differences in the magnitude and sign of the coefficients between the sectoral models. For example, education is a stronger predictor of household consumption in rural and estate sectors than in the urban sector; the lower returns to education in urban areas may simply be a
18
consequence of higher average education levels than in rural and estate sectors.17 Table A7 summarizes the targeting efficiency of the set-up in which the PMT score for a household is assigned depending on the sector it lives in. We see that the under-coverage, leakage, and eligibility rates are largely similar (perhaps there is a modest improvement in undercoverage and leakage) to the results for the baseline model presented in Table 3. This suggests that the differences in coefficients we observe for the sectoral models do not necessarily translate into large improvements in targeting efficiency. The next exercise examines if targeting performance improves when a PMT model is estimated using quantile regression rather than OLS regression. Quantile regression can be an attractive alternative to OLS regression since it is less sensitive to outliers and allows us to calibrate the PMT model to the bottom end of the population distribution that we are typically interested in social protection programs (del Ninno and Mills, 2015; Stoeffler et al., 2015, Brown et al., 2016). Table A8 summarizes the targeting performance of our PMT model estimated using quantile regression at the median and 20th percentiles of the consumption distribution. The cutoff scores for program eligibility are chosen such that total program size is kept the same as in our baseline results in Table 3. We see that undercoverage and leakage rates are very similar between Tables 3 (OLS) and A8 (quantile) and within one percentage point of each other for comparable scenarios. The PMT cutoff scores (which correspond to various percentiles of the true consumption distribution), that yield the same eligibility shares as in Table 3 are lower than those in Table 3. The eligibility share is higher when a PMT model is estimated using quantile regression compared to OLS estimation when we use the same PMT score cutoffs because predicted consumption (thus, lower PMT scores) will also be lower when the model is estimated for lower quantiles.18 Since a PMT model based on quantile regression increases eligibility shares, we need to compare scenarios with similar eligibility shares to get a sense of whether targeting performance would be better than with OLS regression. This finding is consistent with that in Brown et al. (2015), who also find that quantile regression estimated at lower quantiles result in lower undercoverage but at the expense of higher leakage and eligibility rates. Although Stoeffler et al. (2015) implement quantile regression for Cameroon and argue that it gives better results than OLS regression, they do not present evidence of this in terms of performance measures. Next, we experiment with probit and logit estimators to determine the weights of the proxy means test. In particular, we regress the probability that a householdâs per capita consumption falls below the 25th percentile of the welfare distribution, on the same set of characteristics used for the proxy means test. Households are eligible if the predicted probability of being poor is below a particular threshold. This provides an alternative way of determining the weights assigned to household characteristics in the PMT
17 A Hausman test rejects the null hypothesis that the difference in coefficients between the individual sectoral models and the national model is not systematic at the 1% significance level. The relevant Chi-square statistic for the urban, rural, and estate models is -185.11, -366.13, and -83.68, respectively with 36 degrees of freedom. 18 We do not present quantile regression results that show a decrease in undercoverage and an increase in leakage and eligibility compared with OLS regression.
19
model. In practice, using probit or logit to determine the weights makes very little difference, as leakage and undercoverage rates change by less than one percentage point. The final exercise conducts a simple out-of-sample test to examine the sensitivity of the targeting performance to the choice of the sample used to estimate the PMT model. We implement this as a two-fold cross-validation exercise in which we split the sample into two parts (say, A and B). We first estimate the PMT model using sample A, predict consumption for sample B, and compute performance measures for sample B. We then conduct the same exercise with sample B. We finally compare the performance of these two scenarios to the one with the full sample. We assume that the bottom 25% of the population is targeted and the 25th percentile PMT cutoff score is used. Table A9 presents the results of this exercise. Although we only see small differences in the performance measures across the samples, these differences are likely to be aggravated in smaller samples. This exercise is difficult to evaluate since we do not estimate the standard errors of the performance measures. But if Table A9 gives any indication, out-of-sample validation is not a concern if a sample as large as HIES 2016 is used to estimate the PMT model. VI. Conclusion The PMT method has been successfully used to distribute social safety net benefits in many countries. A PMT formula is preferable to a verified means test (e.g. an income criterion) in the Sri Lankan context, because reliable data on income are not available for many workers. The PMT is also attractive because it provides a single index of welfare that can be used for harmonized targeting of several different social welfare programs. Finally, a advantage of the PMT over other targeting methods is its transparency and verifiability. It provides a clear, unbiased determination of a householdâs eligibility whereas other approaches, such as simple means tests and community-level targeting, are more susceptible to reporting errors and imprecise definitions, and may be inconsistently applied in different areas. Despite these arguments for using PMT in Sri Lanka, we recognize that this method of identifying beneficiaries of government transfers may fall short in effectively reducing poverty (Brown et al. 2016). We have estimated and tested three models that parsimoniously predict household welfare from observable and verifiable information. These models have similar targeting performance, but the model that includes characteristics of the household head slightly outperforms those that exclude these characteristics. Any of these proposed PMT formulas would significantly improve the targeting efficiency and coverage of Samurdhi relative to the existing targeting approach, which is based on self-declared income. We expect that an examination of other programs, including elderly support and disability benefit programs, would show similar potential improvements in targeting performance from adopting a PMT. An important issue when considering a PMT in the Sri Lankan context is that Samurdhi eligibility is currently determined separately for each family, whereas the PMT formula is based on a household-level survey. It is not possible to accurately verify whether the definition of the household unit used in the HIES is comparable to that of the family unit used by the Samurdhi program. This is complicated by the fact that neither does a formal definition of a family exist, nor does there appear to be one applied uniformly by
20
Grama Niladharis to select beneficiaries. The household head characteristics used in Model 1 are unlikely to match those of the family head in all cases. Further, even the use of household assets across models may be inaccurate to the extent that household and family assets differ, and family size will be systematically lower than household size. It may therefore be worth considering using the household-level model and adjusting program rules as needed to account for multiple families in a single (eligible) household. Two further considerations are important. First, the choice of eligibility cutoff is key, since it influences both the number of eligible households and the PMTâs targeting performance. A higher cutoff simultaneously increases coverage among the target group and increases leakage of benefits to those outside the target group. The choice of cutoff is ultimately dependent on policy priorities, the size of the budget, and the desired average benefit. Second, it is recommended that the formula should be updated every 6 years to reflect structural changes in living standards and patterns of poverty. Such an update would be in line with the triennial timetable for the HIES. Finally, the basic household level information used to determine household eligibility should be collected and updated more frequently for existing beneficiaries or new applicants, taking into account budgetary and practical considerations, but ideally every two to three years. We recommend a few steps for validating and refining the models presented in this paper. A pilot survey to collect additional data on household characteristics, assets and household consumption could be used to validate and refine the PMT formula. The data could also be used to verify the extent of measurement error in the PMT formula due to potential differences in definition of household and family. This pilot could also be used to validate the PMT model by checking if particular types households are systematically included or excluded (inclusion and exclusion errors). Additional criteria can be proposed for selecting beneficiaries in these scenarios. An important aspect of this pilot should be to verify the definition of the family that is used as the unit of program eligibility. This can be done by collecting information on the family structure within households. In addition to this, Grama Niladharis, who are responsible for selecting beneficiaries, can be enquired for the definition of family they use. In addition to these issues, there are practical aspects of program implementation that we do not address here but may be critical in determining program success. A deep assessment of the current implementation process may help in identifying and addressing sources of implementation errors. Designing a proper grievance or appeals mechanism can help reduce potential exclusion errors. For example, smaller households are less likely to be poor and therefore more likely to receive a higher PMT score. However, this may miss smaller vulnerable households like those headed by single parents or widows with small children. A redress process can help address specific cases of exclusion like these. An effective outreach strategy to enroll the intended population can help in reducing exclusion errors. Similarly, designing an effective exit strategy to remove relevant beneficiaries from the program is crucial in order to reduce inclusion errors and to make the program budget-friendly. Future extensions to our work could also examine the concept of vulnerability threshold, which is set above the targeting threshold and defines a set of household that can be monitored and assisted if needed - i.e. in the event of a shock, the incidence of which can be monitored with rapid phone surveys.
21
Another issue that could be considered relates to the use of administrative data in the PMT process. One possibility is to compare the profile of current Samurdhi beneficiaries from administrative data with those stated beneficiary in a representative household survey. This examination may reveal discrepancies that provide hints about implementation failures. Another possibility is to use administrative data that are aggregated at pre-defined geographic variables. Possibilities include share of households with access to piped water, average classroom size, access to roads, etc. The advantage of including administrative data in the PMT formula would be that it would be less subject to manipulation by households and more timely than household surveys. However, for this to function effectively, mechanisms needs to be in place to ensure that the administrative data is complete, vetted, and is fed into the PMT process in a timely manner. Finally, further study needs to be conducted if PMT can be complemented with additional targeting methods for short-term targeting, for example, to respond to natural disasters. One possibility is to use data from satellite imagery â which can be potentially obtained more frequently than household surveys â to narrow pockets of poverty within disaster-affected areas (Engstrom et al., 2017). The PMT could be subsequently used to identify beneficiaries of emergency assistance within these areas. Another possibility is to potentially incorporate variables derived from remote sensing data in the PMT formula. The advantage of these variables would be that they may be less costly to collect than household surveys, less subject to manipulation by households, and provide more timely information on household welfare. The capacity of government agencies in countries like Sri Lanka may not be currently sufficient to process remote sensing data, but this may change in the due course of time. This may also be an avenue of collaboration between government agencies and development partners.
22
VII. References
Alatas, Vivi, Abhijit Banerjee, Rema Hanna, Benjamin A. Olken, and Julia Tobias. "Targeting the poor: evidence from a field experiment in Indonesia." The American economic review 102, no. 4 (2012): 1206-1240. Alkire, Sabina, and Suman Seth. "Selecting a targeting method to identify BPL households in India." Social indicators research 112, no. 2 (2013): 417-446.
Ayala Consulting. 2002. âFinal Report: Workshop on Conditional Cash Transfers Programsâ Operational Experience.â Paper presented at workshop in Puebla, Mexico, April 29âMay 1
Brown, Caitlin; Ravallion, Martin; van de Walle, Dominique. 2016. A Poor Means Test? : Econometric Targeting in Africa. Policy Research Working Paper;No. 7915. World Bank, Washington, DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/25814 License: CC BY 3.0 IGO.
Castañeda, T. and K. Lindert. 2005. âDesigning and Implementing Household Targeting Systems: Lessons from Latin America and the United Statesâ, Social Protection Discussion Paper Series No. 0526â The World Bank, Washington DC
Coady, D., M.E. Grosh, and J. Hoddinott. 2004. âTargeting of transfers in developing countries: Review of lessons and experienceâ. Vol. 1, World Bank Publications. The World Bank, Washington DC
Del Ninno, Carlo and Bradford Mills, 2015 (Eds.), "Safety Nets in Africa: Effective Mechanisms to Reach the Poor and Most Vulnerable." The World Bank, Washington D.C.
Department of Census and Statistics, Labour Force Statistics Quarterly Bulletin, available at: http://www.statistics.gov.lk/samplesurvey/LFS_Q1_Bulletin_WEB_2016_final.pdf
Department of Census and Statistics and World Bank. 2015. âThe spatial distribution of poverty in Sri Lankaâ, available at: http://www.statistics.gov.lk/poverty/SpatialDistributionOfPoverty2012_13.pdf
Diamond, Alexis, Michael Gill, Miguel Angel Rebolledo Dellepiane, Emmanuel Skoufias, Katja Vinha, and Yiqing Xu. "Estimating poverty rates in target populations: An assessment of the simple poverty scorecard and alternative approaches." (2016). Engstrom, Ryan; Hersh, Jonathan; Newhouse, David. 2017. Poverty from Space : Using High-Resolution Satellite Imagery for Estimating Economic Well-Being. Policy Research Working Paper;No. 8284. World Bank, Washington, DC. Grosh, M., C. Del Ninno, E. Tesliuc, and A. Ouerghi. 2008.âFor protection and promotion: The design and implementation of effective safety netsâ. The World Bank, Washington DC.
Grosh, M., and J. Baker. 1995. âProxy Means Tests for Targeting Social Programs: Simulations and Speculationâ. Working Paper 118, Living Standards Measurement Study, The World Bank, Washington DC.
Grosh, M. 1994. âAdministering Targeted Social Programs in Latin America: From Platitudes to Practice.â The World Bank, Washington DC
Hou, X. (2008) Challenges of Targeting the Bottom Ten Percent: Evidence from Pakistan. Mimeo (Draft), The World Bank, Washington DC
Kidd, Stephen, and Emily Wylde. "Targeting the Poorest: An assessment of the proxy means test methodology." AusAID Research Paper, Australian Agency for International Development, Canberra, Australia (2011).
Klasen, Stephan, and Simon Lange. "Targeting performance and poverty effects of proxy means-tested
23
transfers: Trade-offs and challenges." No. 231. Discussion Papers, Ibero America Institute for Economic Research, 2015.
Narayan, A., T. Viswanath and N. Yoshida. 2005. âProxy Means Test for Targeting Welfare Benefits in Sri Lankaâ, PREM Working Paper Series, Report No. SASPR-7. The World Bank, Washington DC
Sabates-Wheeler, Rachel, Alex Hurrell, and Stephen Devereux. "Targeting social transfer programmes: Comparing design and implementation errors across alternative mechanisms." Journal of International Development 27, no. 8 (2015): 1521-1545.
Sharif, I.A. 2009. âBuilding a Targeting System for Bangladesh based on Proxy Means Testingâ, Social Protection Discussion Paper, No. 0914. The World Bank, Washington DC
Stoeffler, Quentin, Pierre Nguetse-Tegoum, and Bradford Mills. "Generating a system for targeting unconditional cash transfers in cameroon." Effective Targeting Mechanisms for the Poor and Vulnerable in Africa. Washington, DC: World Bank (2015).
Sumarto, S., A. Suryahadi, and L. Pritchett. 2000. âSafety Nets and Safety Ropes: Comparing the Dynamic Benefit Incidence of Two Indonesian âJPSâ Programs.â Social Monitoring and Early Response Unit Working Paper, February
World Bank. 1999. âImproving Social Assistance in Armenia.â Human Development Unit, Country Department III, Europe and Central Asia Region, World Bank Report No. 19385-AM. Washington DC
World Bank. 2003. Armenia: Public Expenditure Review. World Bank Report 24434-AM. Poverty Reduction and Economic management Unit, Europe and Central Asia Region. Washington DC
World Bank. 2016a. âProxy Means Test for Poverty Targeting in Iraq: Technical Noteâ. The World Bank, Washington DC
World Bank. 2016b. âProxy Means Test for Poverty Targeting in Kurdistan: Technical Noteâ. The World Bank, Washington DC
World Bank. 2016c. âSri Lanka Poverty and Welfare: Recent Progress and Remaining Challengesâ, No. 103281. The World Bank, Washington DC
World Bank. 2016d. Sri Lanka - Social Safety Nets Project (English). Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/en/285991480906853560/Sri-Lanka-Social-Safety-Nets-Project
24
VII. Appendix Table A1. Comparison of predictors used for Proxy Means Testing models in South Asia
Variables Sri Lanka Bangladesh Pakistan Present
Study Narayan &
Yoshida
Sharif
Hou Model 1 Model 2 Model 3 (2005) (2009) (2008) HH demographics Household size X X X X X X Dependency ratio X X X Highest education level (split into grade 10, O-level, A-level and degree)
X X X (partial)
Member age X X X Head characteristics Age X X X X Education X X X X Occupation X X X X Marital status X X X X Gender X X X X HH assets Computer X X X X Cooker X X X X Electric fan X X X X X Refrigerator X X X X X Land phone X X X Washing machine X X X Water pump X X Motorcycle X X X X X Car/van X X X X Three-wheeler X X X Four-wheel tractor X X X X Tube well X X X TV X X X Cattle/livestock X X X Bicycle X X Radio/CD or cassette player X Sewing machine X Watch X Air conditioner X Housing quality and facilities Bedrooms per person X X X X X X Type of latrine X X X X X Drinking water source: inside unit X X Electricity for lighting X X X X Have floor tiles/terasso X X X Have walls of brick/kabok/cement X X X X X Have water pump X X Type of roof X X Type of fuel for cooking X X
25
Land ownership/lease/rent X X X Location characteristics X X Community characteristics X Access to remittances X
Table A2. Estimated PMT eligible share and distribution of beneficiaries by actual per capita consumption deciles for different PMT cutoff scores for Models 2 and 3
Quantiles of the true consumption distribution=> Bottom Decile
Bottom Quintile
Top Quintile
Top All
Decile
Model 2
Eligible share
Existing Samurdhi beneficiaries 40.2 37.1 3.8 2.2 18.8 PMT cutoff score*
Score: 878 (20th percentile) 49.3 38.6 0.2 0.2 11.6 Score: 895 (30th percentile) 75.2 65.0 1.0 0.5 23.5 Score: 909 (40th percentile) 89.3 82.2 3.2 1.6 36.1
Distribution of beneficiaries
Existing Samurdhi beneficiaries 21.4 39.5 4.1 1.2 100 PMT cutoff score*
Score: 878 (20th percentile) 42.5 66.5 0.4 0.1 100 Score: 895 (30th percentile) 32.1 55.4 0.8 0.2 100 Score: 909 (40th percentile) 24.7 45.5 1.8 0.5 100
Model 3
Eligible share
Existing Samurdhi beneficiaries 40.2 37.1 3.8 2.2 18.8 PMT cutoff score*
Score: 878 (20th percentile) 46.8 36.9 0.3 0.2 11.1 Score: 895 (30th percentile) 74.5 63.9 0.9 0.4 23.2 Score: 909 (40th percentile) 89.2 82.0 3.5 1.8 36.4
Distribution of beneficiaries
Existing Samurdhi beneficiaries 21.4 39.5 4.1 1.2 100 PMT cutoff score*
Score: 878 (20th percentile) 42.1 66.4 0.5 0.2 100 Score: 895 (30th percentile) 32.1 55.0 0.8 0.2 100 Score: 909 (40th percentile) 24.5 45.0 1.9 0.5 100
See notes for Table 2.
26
Figure A1: Sample size and PMT performance
Figure A2: Distribution of actual vs. predicted consumption for using different sample sizes
27
Figure A3: PMT performance and survey quarter (N=4,500 per quarter)
Figure A4: PMT performance and survey quarter (N=1,500 per quarter)
Figure A5: PMT performance and survey quarter (N=250 per quarter)
0
10
20
30
40
50
60
Undercoverage (%) Leakage (%) Eligibility (%) Poverty rate (%)
Q1 (N=4,500) Q1 (N=4,500) Q1 (N=4,500) Q1 (N=4,500)
Annual (N=4,500) Annual (N=18,000) Annual (N=21,756)
0
10
20
30
40
50
60
Undercoverage (%) Leakage (%) Eligibility (%) Poverty rate (%)
Q1 (N=1,000) Q2 (N=1,000) Q3 (N=1,000)
Annual (N=1,000) Annual (N=4,000) Annual (N=21,756)
0
10
20
30
40
50
60
Undercoverage (%) Leakage (%) Eligibility (%) Poverty rate (%)
Q1 (N=250) Q2 (N=250) Q3 (N=250)
Annual (N=250) Annual (N=1,000) Annual (N=21,756)
28
Table A3. Undercoverage and leakage by province and district* Model 1 Model 2 Model 3
Province / District
Samurdhi Beneficiary Population
Under- cover-
age
Leak-age
PMT eligible share
Under- cover-
age
Leak-age
PMT eligible share
Under-cover-
age
Leak-age
PMT eligible share
Western 9.1 48.6 38.9 11.7 51.4 38.6 11.0 52.2 39.8 11.0 Colombo 5.0 37.6 50.7 9.8 37.0 46.6 9.1 35.1 47.5 9.6 Gampaha 11.9 51.7 30.9 10.4 56.4 30.9 9.4 56.7 32.0 9.5 Kalutara 11.6 51.6 35.4 17.5 54.4 38.6 17.4 57.4 39.9 16.6 Central 17.8 30.2 34.4 33.7 30.1 34.1 33.6 31.7 35.2 33.3 Kandy 19.7 35.2 30.5 27.7 37.2 30.4 26.8 39.3 33.1 27.0 Matale 20.6 35.7 37.4 28.5 36.8 37.6 28.1 39.5 37.4 26.8 Nuwara Eliya 12.4 20.1 37.5 48.5 16.5 36.6 50.0 16.7 36.5 49.7 Southern 21.5 39.4 35.9 22.4 40.9 36.5 22.1 41.8 36.4 21.7 Galle 18.9 37.7 35.2 21.5 39.6 37.0 21.4 40.7 36.5 20.9 Matara 24.2 41.0 27.3 25.7 42.1 27.5 25.3 42.2 28.1 25.5 Hambantota 22.5 39.0 52.2 19.8 41.1 51.6 18.9 43.7 51.7 18.1 Northern 38.1 24.2 37.8 44.7 24.0 38.2 45.1 24.2 37.8 44.7 Jaffna 38.2 20.0 38.5 49.2 20.2 38.7 49.2 21.4 38.3 48.2 Mannar 42.4 30.1 51.3 38.3 26.9 50.6 39.4 24.8 49.4 39.6 Vavuniya 22.6 25.5 54.8 24.5 30.3 56.2 23.7 21.3 53.8 25.3 Mullaitivu 52.9 36.4 23.2 45.3 34.8 26.0 48.2 35.4 25.5 47.4 Kilinochchi 44.7 25.8 25.1 57.9 25.1 26.1 59.2 25.9 26.5 58.9 Eastern 29.8 40.8 29.2 30.9 38.7 30.4 32.6 40.0 29.4 31.4 Batticaloa 34.9 41.8 22.6 36.1 38.9 22.1 37.6 42.1 20.0 34.8 Ampara 26.2 45.7 48.3 23.3 44.4 50.4 24.9 44.3 49.6 24.6 Trincomalee 29.2 35.4 17.4 36.9 33.7 19.5 38.8 33.8 18.9 38.5 North Western 19.4 45.3 44.4 19.5 48.3 44.9 18.6 49.6 44.7 18.0 Kurunegala 22.0 49.9 42.9 18.1 52.0 42.5 17.3 52.5 41.6 16.8 Puttalam 14.0 34.2 46.8 22.3 39.4 49.0 21.4 42.6 49.9 20.6 North Central 18.0 45.8 44.8 20.6 46.3 45.8 20.7 47.3 47.3 20.9 Anuradhapura 18.1 45.4 44.3 19.9 47.5 45.3 19.5 47.8 47.6 20.2 Polonnaruwa 17.6 46.5 45.7 22.0 44.1 46.6 23.4 46.4 46.7 22.5 Uva 20.7 35.9 36.6 36.6 35.7 36.1 36.5 36.5 37.5 36.8 Badulla 15.2 37.7 37.6 36.4 36.9 37.4 36.7 36.0 36.9 37.0 Monaragala 30.5 32.8 34.8 36.9 33.5 33.8 35.9 37.5 38.6 36.4 Sabaragamuwa 24.3 41.9 28.2 29.2 42.4 27.1 28.6 43.6 28.3 28.4 Ratnapura 27.9 36.1 23.6 32.9 36.5 22.8 32.4 38.3 24.3 32.1 Kegalle 19.7 51.5 36.4 24.4 51.9 34.8 23.6 52.4 35.4 23.6
* For target population of bottom 25% and cutoff score at the 30th percentile. Table A4. Undercoverage and leakage by sector*
Model 1 Model 2 Model 3 Samurdhi
beneficiaries Under-
coverage Leakage PMT
eligible share
Under- coverage
Leakage PMT eligible
share
Under- coverage
Leakage PMT eligible
share Urban 11.6 46.4 41.5 12.8 45.8 41.1 12.9 48.0 42.6 12.7 Rural 21.0 40.2 35.7 24.2 41.4 35.8 23.7 42.2 36.1 23.5 Estate 8.4 23.0 33.9 58.3 20.7 33.5 59.7 23.3 34.7 58.7 All 18.8 39.3 36.0 23.7 40.0 36.0 23.5 41.0 36.6 23.2
* For target population of bottom 25% and cutoff score at the 30th percentile.
29
Table A5. Sensitivity analysis: Undercoverage and leakage by household size and age of head, for target population below bottom 25% and cutoff score at bottom 30%
Model 1 Model 2 Model 3 Samurdhi
beneficiaries Under
coverage
Leakage
PMT eligible share
Under coverag
e
Leak age
PMT eligible share
Under covera
ge
Leakage
PMT eligible share
Household size
1 21.5 - - - - - - - - - 2-3 17.3 72.3 39.6 6.4 76.6 40.9 5.5 83.1 37.6 3.8 4-5 18.2 40.7 4- 23.2 40.4 39.4 23.2 40.8 40.5 23.4 6+ 21.7 24.0 30.8 46.4 24.8 31.4 46.3 24.7 31.8 46.6 Age of household head
< 15 0.0 - - - - - - - - - 15-29 11.0 42.5 26.5 22.4 33.0 32.0 28.2 34.3 31.7 27.5 30-45 16.1 37.4 36.2 24.4 32.3 37.6 27.0 32.5 38.2 27.2 45-60 21.3 40.8 36.9 22.4 42.3 37.1 21.9 43.8 37.3 21.4 60+ 19.1 38.9 35.7 24.9 44.9 33.4 21.6 46.3 34.3 21.4
Table A6: Estimated PMT Models by sector
Urban Rural Estate
Variables Coeff. S.E. Coeff. S.E. Coeff. S.E.
HH head characteristics
Female Head 0.02 (0.02) 0.01 (0.01) -0.07* (0.04)
Married -0.10 (0.07) 0.05 (0.03) 0.02 (0.14)
Widowed -0.09 (0.07) 0.04 (0.03) 0.08 (0.14)
Divorced -0.14 (0.12) 0.03 (0.06) 0.22 (0.35)
Separated -0.06 (0.08) -0.02 (0.04) 0.04 (0.16)
Age -0.00** (0.00) -0.00*** (0.00) -0.00** (0.00)
Employment:
Govt/semi-govt employee 0.07*** (0.03) 0.11*** (0.01) 0.04 (0.05)
Private employee 0.06*** (0.02) -0.01 (0.01) -0.01 (0.03)
Employer 0.21*** (0.04) 0.20*** (0.02) 0.00 (.)
HH demographics
Household size (relative to 6 or more): 1 member 0.75*** (0.08) 0.82*** (0.04) 0.83*** (0.12)
2 members 0.53*** (0.04) 0.59*** (0.02) 0.53*** (0.06)
3 members 0.39*** (0.03) 0.41*** (0.01) 0.43*** (0.05)
4 members 0.24*** (0.02) 0.28*** (0.01) 0.30*** (0.04)
5 members 0.15*** (0.02) 0.16*** (0.01) 0.18*** (0.03)
Highest education level of members not currently enrolled:
Grade 10 -0.03 (0.06) 0.11*** (0.02) 0.07* (0.04)
O/L 0.06 (0.06) 0.15*** (0.02) 0.13** (0.05)
30
A/L 0.15** (0.06) 0.23*** (0.02) 0.24*** (0.06)
University degree 0.26*** (0.07) 0.35*** (0.02) 0.43*** (0.12)
Dependency ratio -0.10*** (0.03) -0.12*** (0.01) -0.22*** (0.05)
HH assets (yes=1)
Computer 0.14*** (0.02) 0.11*** (0.01) 0.09* (0.06)
Cooker 0.15*** (0.02) 0.19*** (0.01) 0.18*** (0.03)
Electric fan 0.06** (0.02) 0.13*** (0.01) -0.06* (0.03)
Refrigerator 0.09*** (0.02) 0.11*** (0.01) 0.07* (0.04)
Washing machine 0.15*** (0.02) 0.11*** (0.01) 0.19** (0.08)
Land phone 0.10*** (0.02) 0.03*** (0.01) 0.06** (0.03)
Water pump 0.12 (0.18) 0.14*** (0.02) 0.44*** (0.11)
Motorcycle 0.01 (0.02) 0.13*** (0.01) 0.10* (0.05)
Car/van 0.40*** (0.03) 0.39*** (0.01) 0.37*** (0.13)
Three-wheeler 0.15*** (0.02) 0.14*** (0.01) 0.14*** (0.04)
Four-wheel tractor 0.07 (0.19) 0.20*** (0.03) 0.00 (.)
Housing quality and facilities
Bedrooms per person (DCS defân) 0.23*** (0.03) 0.15*** (0.01) 0.17*** (0.06)
Have floor tiles/terasso 0.16*** (0.02) 0.12*** (0.01) 0.22** (0.10)
Drinking water source: inside unit 0.13*** (0.02) 0.01 (0.01) -0.04 (0.05)
Electricity for lighting 0.04 (0.10) 0.08*** (0.02) 0.16*** (0.06)
Have wall of brick/kabok/cement 0.04 (0.04) 0.01 (0.01) 0.01 (0.04)
Have toilet within unit 0.10** (0.05) 0.05** (0.02) 0.29*** (0.06)
Constant 8.48*** (0.14) 8.32*** (0.05) 8.15*** (0.18)
Observations 3,429 17,394 933
R-squared 0.565 0.514 0.403
Table A7. Targeting efficiency with separate rural/urban/estate PMT models
Cutoff score Under-coverage Leakage Eligible share
Score: 856 (10th percentile) 89.5 12.2 3.0 Score: 868 (15th percentile) 77.4 17.4 6.8 Score: 878 (20th percentile) 63.5 24.1 12.0 Score: 887 (25th percentile) 50.1 29.8 17.8 Score: 895 (30th percentile) 37.9 35.4 24.0 Score: 902 (35th percentile) 27.8 40.6 30.4 Score: 909 (40th percentile) 20.9 45.8 36.5 Note: Model 1 specification was used to generate this table. Target population assumed to be bottom 25% of consumption distribution.
31
Table A8. Targeting efficiency with quantile regression Median Cutoff score Under-coverage Leakage Eligible share
Score: 854 (9.6th percentile) 89.7 11.0 2.9
Score: 867 (14.4th percentile) 77.9 18.7 6.8
Score: 877 (19.3th percentile) 64.3 25.1 11.9
Score: 885 (23.9th percentile) 51.6 30.7 17.5
Score: 893 (28.8th percentile) 39.2 35.7 23.7
Score: 900 (33.5th percentile) 28.9 41.2 30.2
Score: 907 (38.6th percentile) 20.9 46.0 36.6
20th percentile Cutoff score Under-coverage Leakage Eligible share
Score: 826 (2.7th percentile) 89.7 12.3 2.9
Score: 838 (5.0th percentile) 78.2 20.1 6.8
Score: 848 (7.5th percentile) 64.7 25.7 11.9
Score: 856 (10.1th percentile) 51.6 31 17.5
Score: 863 (13.0th percentile) 39.6 36.1 23.7
Score: 870 (16.1th percentile) 29.2 41.3 30.2
Score: 877 (19.5th percentile) 21.2 46.1 36.6 Note: Model 1 specification was used to generate this table. Target population assumed to be bottom 25% of consumption distribution. The cutoff scores were chosen to obtain the same program size as in Table 3.
Table A9. Out-of-sample tests
Cutoff score Under-coverage Leakage Eligible share
Full sample 51.3 30.5 17.5 Half-sample A used to estimate PMT model 52.0 31.9 17.4 Half-sample B used to estimate PMT model 50.5 29.8 17.9
Note: Model 1 specification was used to generate this table. Target population assumed to be bottom 25% of consumption distribution and a PMT score cutoff of 887 (25th percentile).