Social Assistance Pilots Program
SA Pilots Seminar
Hybrid Means Testing (HMT) Model Development
Roman SemkoCASE Ukraine
March, 2010
1.1. Introduction to modelingIntroduction to modeling
2.2. Data analysisData analysis
3.3. Methods for estimationMethods for estimation
4.4. SimulationsSimulations
5.5. Income from assets (agriculture)Income from assets (agriculture)
6.6. Double-blind experiment resultsDouble-blind experiment results
7.7. Model comparisons and conclusionsModel comparisons and conclusions
ContentContent
2
ConceptConcept
3
• The World Bank has developed a methodology for income estimation The World Bank has developed a methodology for income estimation which is based on regression analysis – HYBRID MEANS TESTING which is based on regression analysis – HYBRID MEANS TESTING (HMT)(HMT)
• Under HTM method, eligibility to the SA program is assessed based Under HTM method, eligibility to the SA program is assessed based on the households income modelingon the households income modeling
• Total income is divided into two parts: easy to verify (e.g., pension, Total income is divided into two parts: easy to verify (e.g., pension, stipend) and hardstipend) and hard toto verify (e.g., dividends, shadow wage)verify (e.g., dividends, shadow wage)
• The final goal is to estimate hard to verify share of the income based The final goal is to estimate hard to verify share of the income based on a set of variables, which can be accurately measured and reflect on a set of variables, which can be accurately measured and reflect the hard to verify incomethe hard to verify income
• Hard to verify income is divided into income which is not generated Hard to verify income is divided into income which is not generated by long-term assets (estimated by regression model) and income by long-term assets (estimated by regression model) and income from assets (estimated by formulas)from assets (estimated by formulas)
The main goal of the model is to predict most The main goal of the model is to predict most precisely total family incomeprecisely total family income
Source: Finance Ministry of Ukraine
Data and Knowledge
Methods
Equation which estimates applicant’s income based on the available information:
Y = β1*X1 + β2*X2 + β3*X3 + …
Y X1, X2, X3, …
total income
hard to verify
family structure type and sector of employment education region other
Model Criteria
1. Theoretical validity
2. Simplicity
3. Goodness of fit
4. Significance of explanatory variables
Application and Simulation4
Pilots datasetHBS 2008
Good model should use all available relevant Good model should use all available relevant information for income predictioninformation for income prediction
5
10,622 observations of households with total income
• > 3,000 observations of families with declared income• Cannot be used separately for model estimation since total income is not available
A lot of information could/should be used to guarantee acceptable level of precision
Declared income (DI) is an important indicator in total income (TI) assessment
MATCHING
CharacteristicsTI
CharacteristicsTI DI
Characteristics DI
Pilots datasetHBS 2008
Observations are matched in a way to guarantee Observations are matched in a way to guarantee the highest similarity between themthe highest similarity between them
6
Procedure1. Form groups based on the follow-ing variables: type of settlement, type of assistance, household’s size, # of children, working persons, pensioners, sex of the single-heads household2. Match each observations from HBS to the observation from pilots dataset from the same groups based on the similar characteristics: age of the head, education of the head, etc. using Euclidean distance function3. Each observation from pilots dataset is used for matching no more than 2 times4. Aggregate the groups if there are no good candidate for HBS observation from corresponding group from pilots dataset and match again
Observation 1Observation …Observation K1
Group 1
Observation 1Observation …Observation K…
Group …
Observation 1Observation …Observation KN
Group N
Observation 1Observation …Observation L1
Group 1
Observation 1Observation …Observation L…
Group …
Observation 1Observation …Observation LN
Group N
Total vs. declared income comparison (without SA)
Data comparison: a main difference between HBS Data comparison: a main difference between HBS and pilots applicants occurs in their incomes, and pilots applicants occurs in their incomes, while most of other characteristics are similarwhile most of other characteristics are similar
7
For some regions average income in HBS For some regions average income in HBS significantly differs from the Personal Disposable significantly differs from the Personal Disposable Income (PDI) StatisticsIncome (PDI) Statistics
8
Statistics, PDI HBS
Lviv
LutskRivne
Ternopil Ivano-
Frankivsk Uzhgorod
Chernivtsi
Vinnytsa
Zhytomyr
Chernigiv
Sumy
Kharkiv
PoltavaCherkasy
Kirovograd
Odesa
Kherson
Mykolayiv
Zaporizhzhya
Dnipropetrovsk
Lugansk
Donetsk
Simferopol
Kyiv
Khmelnytsky
Differences in income without SA per capita compared to Chernivtsi region, UAH
– >200 – 100-200 – <100
Lviv
LutskRivne
Ternopil Ivano-
Frankivsk Uzhgorod
Chernivtsi
Vinnytsa
Zhytomyr
Chernigiv
Sumy
Kharkiv
PoltavaCherkasy
Kirovograd
Odesa
Kherson
Mykolayiv
Zaporizhzhya
Dnipropetrovsk
Lugansk
Donetsk
Simferopol
Kyiv
Khmelnytsky
Bayesian econometrics allows combining data Bayesian econometrics allows combining data with aggregated publications of regional PDIwith aggregated publications of regional PDI
9
Standard estimation
Bayesian estimation
Calibration
Researcher artificially determines the model coefficient(s), e.g., if regional macrodata say that income in Kyiv city is 1108 UAH higher than in AR of Crimea, than it is assumed that for Kyiv city applicants income is 1108 UAH higher than for AR of Crimea applicants, other things equal
Coefficients are determined based on the collected observations using standard regression tools (classical econometrics)
Combines both approaches. Estimated coefficient lies between calibrated and estimated in a standard way
Does not lead to significant changes within regions but for regions across Ukraine changes are significant: average predicted income for regions has changes
Linear model is the most simpleLinear model is the most simple
Linear
Des
crip
tion
R2
Pre
dict
ions
58 % (large cities – 65%, small cities – 63%, villages – 48%)
• Linear relation between income and family characteristics• Dependent variable is under the logarithm (log-linear)• Independent variables (IVs) include easy to verify income• Other IVs are: number of children, of working persons, of the elderly, type and sector of employments of household heads, education level
– declared income – predicted income
Concept: the more income the applicant declares, the lower the additional predicted income is – a sort of a “zero sum game”
10
Nonlinear model is performing well when income Nonlinear model is performing well when income differences are high – for the whole HBS sampledifferences are high – for the whole HBS sample
NonLinear
Des
crip
tion
R2
Pre
dict
ions
R2-square is not bounded in [0%,100%] region
• Nonlinear relation between income and family characteristics. The form of relation: cubic or quadratic – since total income sorted in ascending order increases as a polynomial of 2nd or 3rd order• Dependent variable is under the logarithm (log-linear)• Independent variables are as in the linear model
– declared income – predicted income
Concept: as for the linear model
11
Two-step model is effective when there is a large Two-step model is effective when there is a large number of families with zero and nonzero hard to number of families with zero and nonzero hard to verify incomesverify incomes
Two-stage
Des
crip
tion
R2
Pre
dict
ions
47 % (no division by cities)
• At first stage probability that family has shadow income is estimated and then linear relations between income and family characteristics with a hazard of having shadow income is used for estimation • Dependent variable is under the logarithm (log-linear) and does not include salary
– declared income – predicted income
Concept: Stable additional income is added to the declared – “the game with constant markup”.
12
1. DEPENDENT VARIABLE
Informal (shadow) salary was incorporated into the dependent variable (hard to verify income) since it is not easy to verify income
Each model needs a set of adjustments in order Each model needs a set of adjustments in order to become fully usefulto become fully useful
13
Adjustments
3. TIME INCONSISTENCIES
In order to compare incomes across different time period, average growth rates of PDI and its elements were used for time adjustment
2. EXPLANATORY VARIABLES (EVs)
Some EVs which can be used for predictions are hard to verify, e.g., number of mobile phones cannot be accurately measured
4. FAMILY HEADS
The definitions of family heads are standar-dized: male co-head and female co-head are used instead of voluntary definitions
Prediction does not change significantly unless dependent variable is redefined. If the dependent variable is redefined, additional predicted income becomes more stable and decreases with the increase of declared income at a lower rate
Description
Average predicted income exceeds declared by Average predicted income exceeds declared by 2626%%
14
Declared vs. Predicted income (by models)
2727% families will be excluded from the SA % families will be excluded from the SA programsprograms
15
Number of beneficiaries (hypothetical scenario)
Average assistance will drop significantly, except Average assistance will drop significantly, except for low income and fuel subsidiesfor low income and fuel subsidies
16
Average assistance (hypothetical scenario)
Total budget for SA expenditures will decrease by Total budget for SA expenditures will decrease by 27%27%
17
Total expenditures on SA (hypothetical scenario)
New approachCurrent situation
Income from agriculture assets is calculated Income from agriculture assets is calculated based on the developed normativesbased on the developed normatives
18
• Agriculture income is calculated as income per hectar• Normatives are not unified across regions
• Income calculation per hectar and per each animal• Differentiation between cities and villages• Normatives are unified since they are based on the same methodology and data
Calculation procedureInformation, certified by the village/city council
Is not applied to families with disable persons or elderly (>70)
If applicant lives closer than 10 km to the city – apply city normatives
Income from land is a product of land area and normatives
Income from payi is calculated sepatately
Income from lifestock is the product of number of livestock heads times the normative
Average predicted income exceed declared by 28%
New
ap
pro
ach
Cu
rren
t si
tuat
ion
Example of income calculation from agricultureExample of income calculation from agriculture
19
NORMATIVELAND AREA NORMATIVEANIMALS AGROINCOME
CROPS ANIMALS
Only farm-stead area of 0.56 hectars, located in village (Donetsk region)
127.62 per hectar per month
Possess one cow and 10 chickens
Only related through the hayfields and pasturage
Only farm-stead area of 0.56 hectars, located in village (Donetsk region)
412.44 per hectar per month
Possess one cow and 10 chickens (the same)
270.83 for cow, and 4.48 for one chicken
+ 63.81 UAH per month
+ 521.85 UAH per month
Declared Income = 211 UAH
Double-blind experiment: case studyDouble-blind experiment: case study
20
Family description
Father: unemployed and not registered in employment center
Mother: housewife
Age: <18 Age: <3 Age: <3 Age: <3
Commission case and home visit
DENIAL
Model result
LESS thanEligibility threshold = 255 UAH
Model prediction = 308 UAH immediate decision – risky family, need home visit
GRANT?WHILE
MORE thanEligibility threshold = 255 UAH
DENIAL
Cases of SA denials through commission, based Cases of SA denials through commission, based on home inspectionson home inspections
21
Type of assistanceDeclared income
Predicted income
Absolute difference
Relative difference
UAH UAH UAH %Low income 9 133 124 1320Low income 211 308 97 46Low income 264 344 80 30Child care 10 133 123 1238Child care 12 127 114 931Child care 209 286 77 37Child care 250 327 77 31Child care 260 365 105 40Child care 273 330 56 21Single mothers 0 68 68 -Single mothers 222 291 69 31
Predicted income helps to select families for Predicted income helps to select families for home inspectionhome inspection
22
Families selected for inspection Comments• Each family has a chance to be selected for home inspection
• The probability of selections increases as the predicted income is significantly different from declared in absolute and relative terms
Model comparisonModel comparison
23
Characte-ristics
MODEL
Linear (no matching)
Nonlinear Two-step model
Linear (matching)
Linear (Bayesian + matching or not)
Theoretical background
Weak since linear relations are rare in nature
Stronger since takes into account nonlinearities
Strong if selection bias is expected to be
Same as linear (no matching)
Same as linear (no matching)
Simplicity Very simple Simple Slightly complex
Simple in estimation but complex in data matching
Very complex in estimation
R-square High - High High -
Information as an input
Only HBS Only HBS Only HBS HBS and pilots dataset
Very effective use of information
Influence on applicant
High High Medium High High
ConclusionsConclusions
24
Income estimates generated by the models significantly differ from the incomes declared by the SA applicants
Further empirical tests with the models are needed
Initially model results should be used only as an advice rather than a criterion for granting SA benefits
The models may be used as an instrument for selecting families for home inspections4
3
2
1