impact evaluation of performance based contracting for
DESCRIPTION
IMPACT EVALUATION OF PERFORMANCE BASED CONTRACTING FOR. A collaboration between the Ministry of Health, CNLS, SPH, INSP-Mexico and World Bank. GENERAL HEALTH AND HIV/AIDS SERVICES IN RWANDA. Presentation plan. Country Profile Impact evaluation design - PowerPoint PPT PresentationTRANSCRIPT
IMPACT EVALUATION OF PERFORMANCE BASED
CONTRACTING FOR
A collaboration between the Ministry of Health, CNLS, SPH, INSP-Mexico and World Bank
GENERAL HEALTH AND HIV/AIDS SERVICES IN
RWANDA
1
04/21/23
2
Presentation plan
1. Country Profile2. Impact evaluation design 3. Results from the general health
baseline and Follow up4. Results from HIV baseline and
Follow up studies 5. Discussion
2
SECTION 2 :Impact Evaluation
Design
7
04/21/23
8
Study Rationale
No examples of rigorously evaluated bonus payment schemes to public sector health care providers in developing countries
No distinction between the incentive effect and the effect of an increase in resources for the health facilities
Link between worker motivation programs and quality of care
8
TRADITIONAL FINANCING VS PBF Traditional Way of Health financing in Rwanda
was based on 3 sources of finance:
GoR / Founders: Investment cost (Infrastructures & Equipment)
GoR / Private / Population: Recurrent cost (salaries, drugs, materials, trainings,..)
PBF: funds received for remuneration can be derived for enhancing human capacity building, HF materials, or any new strategy aiming at increasing quality.
04/21/23
10
Hypotheses
For both general health services and HIV/AIDS services, we will test whether PBC:
Increases the quantity of contracted health services delivered
Improves the quality of contracted health services provided
Does not decrease the quantity or quality of non-contracted services provided,
Decreases average household out-of-pocket expenditures per service delivered
Improves the health status of the population
10
04/21/23
11
Evaluation Design Make use of expansion of PBC schemes over time The rollout takes place at the District level Treatment and control facilities were allocated as
follows: Identify districts without PBC in health centers
in 2005 Group the districts in “similar sets” based on
characteristics: rainfall population density livelihoods
Flip a coin to assign districts within each “similar” to treatment and control groups. 11
12
04/21/23
13
Roll-out plan Phase 0 districts (white) are those districts in
which PBF was piloted Cyangugu = Nyamasheke + Rusizi districts Butare = Huye + Gisagara districts BTC = Rulindo + Muhanga + Ruhango + Bugesera + Kigali
ville
Phase 1 districts (yellow) are districts in which PBF is being implemented in 2006, following the ‘roll-out plan’
Phase 2 districts (green) are districts in which PBF is not yet phased in; these are the so-called ‘Phase 2’ or ‘control districts’ following the roll-out plan. According to plan, PBF will be introduced in these districts by 2008. 13
Baseline Health Facility SamplePhase I
(Intervention district)Phase II
(Control districts)
DistrictTotal Facilities District
Total Facilities
Kibungo 11 Kirehe 10Nyanza 2 Kamonyi 9Gakenke 11 Rwamagana 3 Nyagatare 13Gatsibo 8 Kayonza 11 Nyamasheke 2 Karongi 17Ngororero 6 Rutsiro 10 Nyabihu 10Nyaruguru 9 Nyamagabe 13Burera 10 Musanze 11Total 83 Total 83
TOTAL 166 HEALTH FACILITY
04/21/23
16
Program Implementation Timeline
Jan-06 Mar 06 Jun-06 – Sept 06 2007 Feb-08-Apr 08 Apr-08
Phase I Start of intervention
Program Implementation
Phase II
Start of intervention
Impact Evaluation
SURVEYS Baseline
(General Health) Baseline
(HIV/AIDS)
Follow-up
16
More money vs. More incentives Incentive based payments increase the
total amount of money available for health center, which can also affect services
Phase II area receive equivalent amounts of transfers average of what Phase I receives Not linked to production of services Money to be allocated by the health center Preliminary finding: most of it goes to salaries
18
The baseline has 4 surveys
General Health facility survey (166 centers)
General Health household survey (2,016 HH)
HIV/AIDS facility survey (64 centers) HIV/AIDS household survey (1994 HH)
19
Baseline Field Sampling: GH HH Randomly selected FOUR CELLS per HF
Done in SPH using Epi Info and given to team leaders Randomly selected THREE ZONES per cell
Done by GH HH team leader in field Obtained household lists for each of the zones and
randomly selected ONE HOUSEHOLD per zone Done by GH HH team leader in field with zone/village
leader Random sample of 12-13 households with children
< 6 years old per HF 2,159 households.
Baseline Field Sampling: GH HH
HEALTH FACILITY
CELL 1 CELL 2 CELL 3
HH 1
CELL 4
HH 4HH 3HH 2 HH 6HH 5 HH 7 HH 9HH 8 HH 10 HH 12HH 11
Validity of Sample
Require two different validations: Validate the sampling for the evaluation design
Diff in means tests between Phase I and Phase II to determine if intervention and comparison groups balanced at baseline
Validate the quality of data Compare descriptive stats to other sources of national
data (i.e.: 2005 & 2007 DHS, MOH data)
04/21/23
23
Analysis PlanAll analyses will be clustered at the district
level Compare the average outcomes of facilities
and individuals in the treatment group to those in the control group 24 months after the intervention began.
Use of multivariate regression (or non-parametric matching) : control confounding factors
Test for differential individual impacts by: Gender, poverty level Parental background (If infant : maternal
education, HH wealth) 23
04/21/23
24
Difference in differences models To test the robustness of the analysis Control sample (both observed and unobserved)
heterogeneity A two-way fixed effect linear regression:
Where:
is an outcome variable for facilitiy i in period t
or for individual i who lives in the catchment area of facility i,
is an indicator of whether facility i
it it k itk i t itk
it
it
y PBC B X
y
PBC
is being paid by PBC in period t,
are time varying control variables,
i is a facility fixed effect,
is a year fixed effect,
is an error term.
itk
t
it
X
24
SECTION 3 :Descriptive Results from General Health Baseline
25
GENERAL HEALTH FACILITIES
BASELINE SURVEY
26
27
General Health Centers Survey:Content
All 166 centers in General Health General characteristics Human resources module: Skills,
experience and motivations of the staff Services and pricing Equipment and resources Vignettes: Pre-natal care, child care,
adult care VCT, PMTCT, AIDS detection services
Exit interviews: Pre-natal care, child care, adult care, VCT, PMTCT
Classification of facility 28
Baseline Health Facility:Main sources of revenue
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Primary Source
Secondary Source
Tertiary Source
Baseline Health Facility:Human Resources
On average:
1 doctor for every 31,190 individuals,
1 nurse for every 4,835 individuals
30
Obs Mean Std. Dev Min MaxNurse A1 165 0.18 0.45 0 3Nurse A2 166 6.31 8.32 0 70Nurse A3 164 0.29 0.65 0 4Auxilliary Staff >1yr 166 1.14 1.61 0 12Auxilliary Staff <1 yr 161 0.78 1.27 0 6Laboratory Technician 165 0.89 1.65 0 20Full-time Support Staff 166 3.87 3.07 0 28
Baseline Health Facility: Services
Category Service% Facilities
Providing Service
Avg. Years Providing Service
Avg. Charge for Service (excluding registration fee)
BCG 98.81% 16.78 -DPT 98.81% 11.91 -Polio 98.81% 16.51 -Measles 98.81% 16.4 -Tetanus 98.21% 16.48 -Children <5 yrs 100% 20 RWF 158.99Adults 100% 19.76 RWF 164.48Child Grow th Monitoring 86.31% 18.03 RWF 16.46Nutritional Information 92.81% 16.83 -Prenatal Care 98.21% 15.51 RWF 35.31Normal Delivery 89.22% 18.6 RWF 717.76Ceasarian Delivery 1.95% 30.75 RWF 892.86Assisted Delivery 9.03% 17 RWF 513.64Home Delivery 14.19% 11 RWF 41.67Blood Transfusion 2.58% 36.5 -Pill 84.05% 10.96 RWF 10.23Injection 82.82% 10.72 RWF 12.59Implant 18.47% 12.2 RWF 85.29Male condoms 71.78% 9.05 RWF 2.78IUD insertion 12.74% 17.17 RWF 13.04Female sterilization 0.64% . -Male sterilization 0.64% . -Diagnosis 36.36% 10.17 RWF 59.09Treatment 49.70% 10.51 RWF 6.63Malaria Treatment 97.56% 17.01 RWF 332.91
Family Planning
TB
Curative Care
Immunizations
Summary of Services Provided
Delivery Services
Baseline Health Facility: Lab Tests
Number of Obs. cstd
Number of Obs. cstd
Anemia 80 37.50% 0.487 84 29.76% 0.460 -1.046
Pregnancy 80 56.25% 0.499 84 34.52% 0.478 -2.846
Blood Count 80 1.25% 0.112 84 3.57% 0.187 0.960
Urine dip 80 50.00% 0.503 84 60.71% 0.491 1.380
Stool Sample 80 8.75% 0.284 84 3.57% 0.187 -1.385
Parasites 77 89.61% 0.307 84 94.05% 0.238 1.029
Malaria 80 50.00% 0.503 84 53.57% 0.502 0.455
Malaria dip 79 55.70% 0.500 84 72.62% 0.449 2.277
TB 80 36.25% 0.484 84 40.48% 0.494 0.553
HIV 80 62.50% 0.487 84 55.95% 0.499 -0.849
Gonorrhea 80 18.75% 0.393 84 21.43% 0.413 0.425
Syphilis 80 10.00% 0.302 84 8.33% 0.278 -0.368
Pap smear 80 38.75% 0.490 84 36.90% 0.485 -0.242
Lab Tests
Phase I (Intervention) Phase II (Control)
T-Stat
Baseline Health Facility:Overall Patient Satisfaction
4.08%
4.83%
14.66%
76.44%
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00%
Very Unsatisfied
Not Satisfied
Very Satisfied
Satisfied
Baseline Household Data: Education37
Sample: ages 6 and up
Variable Nr. Obs Mean Std. Error Nr. Obs Mean Std. Error T-stat
Ever attended 3104 79.68% 0.011 4042 80.88% 0.010 -0.825
No schooling 2458 10.26% 0.007 3252 10.88% 0.007 -0.606
At least some Primary 2458 82.93% 0.010 3252 79.97% 0.010 2.195 **
At least some Secondary 2458 6.46% 0.008 3252 8.42% 0.007 -1.841 *
Attended school in last12 months 2391 41.13% 0.015 3181 41.34% 0.012 -0.112Able to read Kinyarwanda 2532 63.59% 0.015 3359 66.37% 0.014 -1.375
PHASE I (Intervention) PHASE II (Control)
Baseline Household Data: Assets
Variable Nr. Obs Mean Std. Error Nr. Obs Mean Std. Error T-stat
Complete sofa set 921 2.18% 0.005 1147 3.08% 0.007 -1.073
Radio 921 49.63% 0.023 1147 50.27% 0.017 -0.221
Radio-cassette or music system 921 4.29% 0.009 1147 5.89% 0.008 -1.274
Telephone Mobile 921 1.07% 0.005 1147 3.50% 0.007 -2.778 ***
Mosquito nets 921 22.53% 0.023 1147 28.00% 0.027 -1.542
Sewing machine 921 0.76% 0.003 1147 1.18% 0.004 -0.883
A bed 921 62.82% 0.023 1147 58.71% 0.022 1.290
Wardrobe 921 4.53% 0.009 1147 6.07% 0.010 -1.155
Metalic library 921 0.57% 0.002 1147 1.28% 0.004 -1.645
Table 921 63.47% 0.022 1147 63.40% 0.021 0.021
Chair 921 85.48% 0.018 1147 84.82% 0.019 0.255
A bicycle 921 16.37% 0.020 1147 17.26% 0.027 -0.262
PHASE I PHASE II
Baseline Household Data: Prenatal Care
40
Variable Nr.Obs. Mean Std.Error Nr.Obs Mean Std.Error T-stat
Of all births since Jan. 2005
In facility birth 1238 32.05% 0.026 1462 34.87% 0.025 -0.777
Of most recent pregnancy
Times received PNC 723 2.781 0.046 900 2.687 0.052 1.368
Injection to prevent tetanus 728 74.52% 0.023 900 72.56% 0.019 0.654
Phase I Phase II
Baseline Household Data: Child Immunization
42
Variable Nr.Obs. Mean Std.Error Nr.Obs Mean Std.Error T-stat
12-23 months oldfully_immunized 262 75.36% 0.03 295 77.04% 0.04 -0.33
12-71 months oldfully_immunized 1154 63.77% 0.03 1387 65.28% 0.02 -0.43
Phase I Phase II
Baseline Household Data: Child Recent Illness and Symptoms (<6 years)
30.79%
20.65%
16.37% 15.85%
11.71%
7.16%
2.77% 2.20%
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
Fever Respiratory Diarrhea Headache Malaria Eye Infection Ear Infection Accident
Baseline Household Data:Child Biomarkers (<6 years)
18.67%
7.51%
22.16%
27.82%
19.53%
7.33%
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
Anemic Malaria Fever and Malaria
Phase I
Phase II
Validity of Sample and Data46 Evaluation design
Of 110 key characteristics and output variables of HF,
the sample is balanced on 104 of the indicators.
Of 80 key HH output variables, the sample is balanced
on 73 of the variables. Quality of data
HH Results comparable to the 2005 DHS, MOH data
Follow-up Field Sampling: GH HH Result:
In total 2159 were suppose to be surveyed in the catchment area of 167 facilities.
1888 (87%) were interviewed in 2006 and 2008 267 (12%) were replaced 4 (0.2%) were not found and not replaced
% of replacement by region: South (24%), North (14%), East (4%) and West
(13 %)
HIV HEALTH FACILITIES BASELINE SURVEY
53
Baseline Health Facility Sample
Phase I Phase II MAP TOTAL SAMPLEFacility Level Data HIV/AIDS Services and Human Resources 18 18 28 64Provider Level Data General Providers 18 18 28 64 PMTCT Providers 17 17 27 61 VCT Providers 17 18 27 62TOTAL 52 53 82 187Patient Level Data PMTCT Patients 108 108 107 323 VCT Patients 116 87 209 412TOTAL 224 195 316 735
Assessment of the quality of ARV and HIV/AIDS service delivery: Health Facility Sample
Baseline Household Sample
38 ARV facilities for household sampling 13 MAP; 10 Phase I; 15 Phase II ARV sites for HH sampling
Original sample of 1,487 patients from health facilities and associations
Final sample consists of 1,961 households and 7,494 individuals
Patient Treatment: Distribution
Number %
No Treatment 196 13.18%
Cotrimoxazole 439 29.52%
ARV 852 57.30%
TOTAL 1487 100.00%
Household Level: Sample Size of Patients
We find there is an additional sample of individuals in our household sample who self-report HIV+ status
Based on this, final distribution of patients:
Patient Sample: Distribution
Number %
Phase I 701 39.47%
Phase II 694 39.08%
MAP 381 21.45%
TOTAL 1776 100.00%
Baseline Health Facility:Main sources of revenue
Obs Std Obs Std tstat
funding_centralgovt 17 11.76% 0.33 18 33.33% 0.49 1.53
ratio_centralgovt 16 6.25% 0.25 17 21.76% 0.35 1.45
funding_province 17 23.53% 0.44 17 5.88% 0.24 -1.46
ratio_province 17 10.67% 0.25 16 1.42% 0.06 -1.42
funding_healthdistri 17 0.00% 0.00 17 5.88% 0.24 1.00
ratio_healthdist 17 0.00% 0.00 16 1.67% 0.07 1.03
funding_admindistric 17 11.76% 0.33 17 0.00% 0.00 -1.46
ratio_admindist 17 2.15% 0.06 16 0.00% 0.00 -1.38
funding_userfees 17 82.35% 0.39 17 70.59% 0.47 -0.79
ratio_userfees 16 28.07% 0.37 16 27.11% 0.32 -0.08
funding_mutuelles_cu 17 52.94% 0.51 18 61.11% 0.50 0.48
funding_mutuelles_me 17 64.71% 0.49 18 61.11% 0.50 -0.21
ratio_mutuelles 17 50.48% 0.41 17 42.38% 0.40 -0.59
donsgov 17 4586525 3057811.00 16 1.55E+07 35600000.00 1.25
donsnongov_dum 6 100.00% 0.00 12 75.00% 0.45 -1.33
donsnongov 12 7974796 14100000.00 14 8591750 7649821.00 0.14
comite_gestion 18 33.33% 0.49 18 44.44% 0.51 0.67
comite_sante 18 55.56% 0.51 18 44.44% 0.51 -0.65
other_management 18 11.11% 0.32 18 11.11% 0.32 0.00
General Facility Information: Finances
Phase IIPhase I
Baseline Health Facility: Services
HIV/AIDS Services Offered
% Offer Service
HIV counseling 96.83%
Rapid HIV test 91.8%
Elisa HIV test 7.32%
CD4 Count 12.73%
Viral Load 2.5%
Systematic Testing of Blood Donations 19.51%
VCT 88.14%
Distribute Condoms 93.22%
PMTCT 93.10%
Preventative Cotrimoxazole Treatment 100%
Treatment of OIs 91.8%
ARV 80.7%
TB Diagnosis 84.75%
TB Treatment 92.98%
Nutritional Support 80.85%
Baseline Health Facility:Overall Patient Satisfaction
6.05%
0.47% 1.40% 1.14%
58.60%
49.77%
54.88%
48.86%50.00%
43.72%
49.77%
35.35%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Time Attended Confidentiality Staff Attitude Cost medications
Not satisfied
Satisfied
Very Satisfied
HIV Patients vs. Non-Patients:Mortality in the Household
Death in the last 5 years
7.61%
16.84%
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
14.00%
16.00%
18.00%
Patients Non-Patients
Patients vs. Non-Patients:Mortality in the Household in the last 5 years
35.08%
71.88%
15.79%
94.09%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Illness related Death Sold Assets
Patients
Non-Patients
HIV Patients vs. Non-Patients:Health Care Utilization
Illness in last 6 months
69.67%
17.32%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Patients Non-Patients
Patients vs. Non-Patients:Sexual Behavior
39.63%
76.74%
56.14%
37.42%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
Sex in the last 12 months Condom during last encounter
Patients
Non-Patients
Follow-up Field Sampling: HIV HH Objective:
Return to the same households to create panel data set (2006-2008)
Print baseline roster (names, codes) and keep consistent across waves
Account for household members who left and new arrivals from 2006-2008
Follow-up Field Sampling: HIV HH Result:
In total 1996 were to be surveyed in the catchment area of 38 ARV facilities.
1,716 (87.06%) were interviewed in 2006 and 2008
280 (12.94%) were not found and not replaced % of missing by region:
South (12.6%), North (12.7%), East (7%) and West (16.4%)