Social Assistance Pilots Program

SA Pilots Seminar

Hybrid Means Testing: An Overview

Janusz Szyrmer March 2, 2010

Our assignment:

As part of Pilot #3: Carry out experiments on developing new means

testing methods in Ukrainian conditions, with a focus on the Hybrid Means Testing (HMT) methodology, as pioneered by the World Bank– Investigate several variants of HMT– Test applications of the HMT methodology in five pilot

offices; the results of the experiment will not be used to decide on the disbursement of actual social benefits to applicants

– Use the outcomes of the experiments to draw conclusions on the applicability of the method in Ukraine

2

In this presentation:

Social Assistance Performance HMT Concept• Verified Means Testing• Proxy Means Testing• Randomized Means Testing • Self Selection• Income/Non-income Testing

HMT Procedure Experiments with HMT HMT Applications 3

SA Performance

The performance of Ukraine’s current SA system is satisfactory in terms of targeting the recipients of SA benefits, subjected to means testing.

• Typically verified means (income) testing is employed. • Also other methods are used (proxy means testing, unverified means

testing, randomized means testing, and self-selection),but somewhat irregularly/haphazardly, often without clear rules, sometimes simply due to limited capacities (overburdened SA office staff).

As concerns total SA transfers (all benefits and privileges, combined, in

Ukraine), the targeting performance is inferior, due to a lack of appropriate rules and physical capacities for subjecting more benefits/privileges to the rigors of means testing.

4

HMT Concept (1)

HMT belongs to instruments used in social assistance that:• Target those who need SA the most (by reducing the errors of inclusion

and exclusion), and• Support the efficient management of scarce budget resources

HMT is intended to:• Take advantage of modern means testing approaches• Use them broadly and flexibly• Design the best bundle of methods appropriate to each SA application• Improve efficiency of means testing methodology

5

HMT Concept (2)

HMT is a “hybrid” because it involves both direct and indirect methods, typically:

• Direct: Verified income testing: formal verification of incomes• Indirect: Proxy means testing: Incomes and/or needs assessment

based on “indirect” observable/measurable indicators that are related to or correlated with the conditions/incomes they reflect, including place of residence and socio-economic characteristics

Other means testing approaches/methods include:• Randomized means testing• Self-selection / self-verification• Also in some cases: Unverified means testing

6

Verified Means TestingUses formal income and asset tests to determine applicants’ eligibility for SA.Involves checking payroll data, pensions, SA benefits, sometimes also

banking statements, vehicle documentation, etc.In many cases the verification is automated and includes information about

tax records, unemployment data, etc. Is effective in countries with a small shadow economy where a large share of

incomes, expenditures and wealth are formal, monetized and well-documented. Extensive verification of information promotes transparency and credibility (provided that it is conducted in a standard way with equal treatment of all applicants).

Tends to be costly to implement, and both administratively and technically difficult in CIS countries with high degrees of informality in labor markets. It is widely used in the USA.

7

Proxy Means TestingMay involve:- Formal verification of proxies (possession of a car and other financial

and non-financial assets)- Application of econometrically estimated coefficients from a model,

typically a regression model, principal components analysis, or other methods

- Use of accounting formulas which derive “imputed income”, based on some research results, e.g., in agricultural economics

- Assigning scoring/grades to living conditions, SA needs, etc.- Use of “reliability weights” assigned to different pieces of information, in

order to generate a summary score (e.g., the Bayesian method presented later during this session)

8

Randomized Means Testing May be used as a cheap variant of VMT. It may deliver reasonable accuracy,

while requiring less effort in verification of household incomes. Instead of all applicants, only a randomly selected small sample of households is thoroughly verified.

The mathematical/statistical sciences and extensive practice provide a solid base for the selection of random samples of households to be verified, in accordance with rigorous procedures (the samples cannot be selected arbitrarily by the SA staff).

If the procedure is not distorted by some irregularities, then this method may be considered the best low-cost method which is fair, transparent and credible. Its effectiveness may be strengthened by enforcement of some sanctions for provision of faulty information.

9

Self SelectionIs widely used in many countries. People select themselves for SA and may

simply decide not to apply for SA, if they believe that they are not eligible. It is facilitated by one or more of the following factors:– Availability of accurate information to population, based on which persons who do

not qualify for SA or whose chances for getting an SA are very low do not apply– Existence of sanctions for providing inaccurate information or social pressures which

discourage fraudulent applications– Low value of SA (limited to a minimum in-kind or financial support) combined with

high effort (time consuming) application procedures which discourage persons with higher incomes or those who could use the time needed for SA applications to earn money in a different way

Self-selection is a relatively low cost method, but may be considered unfair, it may result in a low targeting accuracy (a high error of exclusion – high number of those who do not receive benefits despite being eligible). 10

Income/Non-income Testing

11

HMT Procedure (1) HMT in the format developed by the World Bank is tasked with estimation of

total family income.Total income is broken down to two categories:

(i) easy-to-verify incomes (EVI) such as official wages, pensions and allowances and

(ii) hard-to-verify incomes (HVI) derived for instance from self-employment, land plot use or informal activities.

Total income is calculated as a sum:EVI + HVI

EVI is reported by an SA applicant and formally verified by SA office.HVI is imputed by social inspectors or other SA workers on the basis of HVI

indicators, derived from an econometric model or by means of other methods. 12

HMT Procedure (2) SA applicants provide information on selected personal and household characteristics which

are found to be important determinants of incomes, are easy to report and verify (income indicators).

Indicators include, for instance, demographic and social characteristics of an SA applicant and his/her family members (such as sex, age, education, family status, etc.), work related data (employment status and economic sector), data on possession of selected assets (e.g., a car), data on dwelling and utilities (surface, # of rooms, equipment), on utility bills, on land plot and livestock, as well as on location (oblast, big city, small urban area or rural area).

Each indicator has a prescribed coefficient (weight) converting it into a “portion” of imputed income.

Applying these coefficients, in accordance with some formulas, and summing the “portions” up generates a total amount of hard-to-verify income

Adding easy to verify income to hard to verify income generates (predicted) total family income

13

Experiments with HMTExtensive experiments with HMT are needed in order to design optimal rules and

procedures: Experiments with both income tests and non-income tests (e.g., an augmented

“Inspection Act” proposal) Experimental introduction of proxy means tests, as

1. Additional information source2. An instrument for “Client profiling”3. Extension of currently used methods (rather than their replacement),

introduced gradually in the format of biding pilots (i.e., affecting grants of SA benefits), only in selected regions, applied only to certain categories of applicants, such as those who:• suffer from chronic (long-term) poverty • do not have formal income sources or reside in regions with limited opportunities

for formal employment• generate most of their incomes from agriculture or from the ownership of non-

farm assets

14

HMT Applications (1)The hybrid nature of HMT enables one to combine together a few approaches in order to

establish an “optimal bundle” of methods satisfying a number of conditions, in particular:– Political acceptability and fiscal feasibility: HMT must provide results that are feasible

in both political and fiscal terms. – Simplicity, harmonization, low cost and administrative feasibility: HMT must be as

simple as possible and keep the cost of SA administration at a low level.– Technical feasibility: In the economies with a large informal sector a verification

limited to formal incomes fails to assess full household incomes. An HMT method, combining formal and informal income sources, is needed.

– Level of transparency and fairness: The assessment methods should meet clarity and social justice requirements.

Eligibility assessment mechanisms should seek to maximize targeting accuracy at an acceptable (low) cost and in a transparent manner while satisfying political/policy requirements and fairness (social justice) conditions.

15

HMT Applications (2) There are several reasons for which HMT should be considered for

applications for the needs of the Ukrainian SA system:– International experience shows that this method may generate

satisfactory targeting results. E.g., about 85% of the benefits of HMT programs in Chile and Mexico are received by the poorest 40% (two lowest income quintiles) of households in those countries which is considered a good targeting.

– There is a considerable international experience of applying HMT for SA targeting, which may be used in Ukraine.

– It is more effective in the countries where a relatively high share of incomes is generated by the informal sector.

16

HMT Applications (3)HMT, relying on income proxies, possesses some limitations which must be considered in

making choices for income estimation:– The results generated by a regression model are based on a sample which while

providing reasonable approximations remains a random set of data which is always subject to some distortions. For various reasons people never provide fully accurate information (due to mistakes, unwillingness to disclose information, etc.). In all household surveys worldwide income data tend to be underreported.

– The models used data from a period during which the survey was run. For Ukraine it was the year 2008. The data do not reflect income and price changes after 2008. Also, the models may fail to reflect the situation of a household in which there have occurred some recent changes of income.

– The static nature of the model requires continuous collection of data on income, consumption and assets, with which the models must be updated. This may become quite a challenge for SA staff required to keep recalculating the proxy income figures of all applicants/beneficiaries.

– The income predictors generated by the HMT regression models are averages. While many households are likely to have incomes equal or similar to these averages, in some cases the actual incomes may be significantly different.

17

HMT Applications (4)

The HMT-type methods are promising but their introduction may require a longer period. This can be facilitated by a number of possible improvements:– The use of budget funds: A detailed analysis of the costs and efficiency of

the SA system is needed, which may lead to efficiency enhancements. – National database: A systematic data collection tailored to HMT needs

should be initiated and data exchange among relevant governmental agencies and other institutions should be strengthened.

– Analytic capacities: Specialized analytical units in MLSP and local SA offices should be established and equipped with appropriate hardware and software. A professional training needs to be arranged.

– Randomized means testing: Application of modern statistics methods may be introduced, supported by appropriate guidelines and regulations from the MLSP.

18