From the Concept Note to

realities on the ground:

Challenges and lessons from

implementing RBF impact

evaluations

Stages of impact evaluation

Concept Note

Finalize Program

and IE Design

IE Baseline

Implement RBF

IE Endline

Analysis and Dissemination

Central African Republic

Cameroon

Benin

India

Impact of the crisis on the

Implementation of the impact

evaluation in CAR: Phase: Baseline survey

Gervais Yama

Plan PBF pilot project in Central

African Republic • PBF in public, confessional and NGO facilities in 9 prefectures in

Regions 2, 3, 4, and 6

• A population of approximately 2.5 million inhabitants

4

Impact evaluation research questions

• The impact evaluation will focus on the

following primary research questions: o Does varying the level of PBF payments for MCH services

delivered lead to variations in MCH service coverage and

quality outcomes?

o Or can similar results be achieved at lower cost?

o What is the optimal fee schedule (level of PBF payments)

for improving MCH service coverage and quality?

5

IE intervention groups

T1: Complete PBF package C1: Complete PBF but with

payments at 50% of PBF

payment levels in group T1

C2: Statut quo

6

Implementation of the baseline survey

7

• The impact evaluation baseline survey was implemented in two

phases:

(i) The health facility survey was conducted in March 2012 and the

household survey was conducted in November-Decembre 2012;

(ii) 351 health facilities and 5464 households were interviewed

during the baseline surveys.

Impact of the crisis on implementation

8

• Three events have affected indicators that were to

be used for the impact evaluation baseline survey:

(i) The first is due to the offensive armed groups in December

2012, just at the end of the collection of the baseline survey;

(ii) The second is in March 2013 with the coup by armed groups

that led to the departure of President Bozizé;

(iii) The third event began on December 5, 2013 by another armed

group, the "anti-Balaka."

Are the data collected still valid for the

impact evaluation?

9

At the health facility level

• the fleeing of health center staff to Bangui;

• looting of drugs and equipment in the majority of health

facilities;

• Very few facilities are still functional, most are those that

receive support from NGOs such as Doctors Without Borders;

• The level of most of the indicators obtained in the 2012

survey deteriorated at all levels of the health pyramid,

especially in the project and IE zones, which are the areas

that were affected by attacks by armed groups.

10

At the household level

• The crisis has deteriorated living conditions of households:

Problems with access to food, pillaging granaries and farming

areas;

• Homes were ransacked or burned with property taken by

armed groups;

• Non-existence of certain villages due to entire communities

fleeing the countryside due to attacks from armed groups;

• Susceptibility to disease due to inaccessibility to health

services on the one hand and poor living conditions on the

other.

Are the data collected still valid for the

impact evaluation?

What are the implications for

implementing PBF?

11

• With the current state of things, how will PBF be

implemented vis-à-vis the crisis? Given that:

o At health facilities, the majority are not operational, and the few

supported by NGOS such as MSF do not have enough personnel,

equipment and drugs?

o At the household level there is a large change in the living standards of

the population affected by the crisis?

Thank you for your kind attention

12

Challenges and lessons from

implementing an RBF impact

evaluation in Cameroon Phase: Preparation and implementation

Jospeh Shu Atanga

INTERVENTION

ZONE

CAMEROUN CAMEROON

Impact Evaluation Implementation

Milestones

• Public Launching of the project

• Public Randomization of health facilities into the intervention and control groups with predefined characteristics

• Identification of all the communities and legalized health facilities in the catchment areas

• Mapping of the communities served by each health facility

• Baseline data collection (health facilities and household surveys

• Training of different stakeholders on PBF

• Drawing up of business plans and signing of different types of contracts

• Production and declaration by different health facilities

• Verification and validation by Purchasing Agency

• Payment to facilities as required

T1

Complete PBF with

medical personnel

performance bonuses

C1

PBF with subsidies

unrelated to

performance

C2

Only enhanced

supervision as in Q1 and

C1.

C3

Status quo: no form of

intervention

Survey basis EI and

follow-up survey

Yes Yes Yes Yes

Contract Classical contract for PBF Contract stipulating the

conditions of PBF for

control / audit and

supervision

Contract providing technical

support in terms of

enhanced supervision

No contract

Business plans Yes to be developed Yes to be developed Business Plan light, based

on the enhanced

supervision

No Business plan

Evaluation of the

quality

Quality assessment,

feedback and taken into

account in the payment.

Quality assessment with

feedback as in T1, but no

effect on payment

Evaluation of the quality

with feedback as in T1.

No quality assessment

Declaration/verificati

on of the quantities

of services

Declaration and

verification of the

quantities of services

produced

Declaration and

verification of the

quantities of services

produced as in T1

services

Declaration and verification

of the quantities of services

produced as in T1 services.

No survey or audit

Payment Linked to the performance Payment unrelated to performance

No payment No payment

Management

autonomy

Managerial autonomy with

retention of all proceeds

Managerial autonomy

with retention of all

proceeds

No autonomy of

management, maintenance

of the existing ‘quote-part’

(incentives) system.

No autonomy of

management,

maintenance of the

existing ‘quote-part’

(incentives) system.

Monthly activity

report submitted to

the health district

Yes Yes Yes Yes

Major Lessons

• Different Intervention and control groups o Payment of subsidies (T1 vs C1)

o Management autonomy T1, C1 vs C2,C3

o Enhanced Supervision T1,C1,C2 vs C3

• Identical Characteristics for intervention and control groups during randomization o Urban vs rural

o Public vs private

• Equal payments for the T1 and C1 groups o T1 health facilities paid based on production

o C1 health facilities paid based on % population contribution to the total population of all the C1 facilities

• Close monitoring of the veracity of the declared services and community perception of quality of care.

• Mix of baseline/endline and routine data collection to inform the M&E

• Tracking of the use of the subsidies by the final beneficiary health facilities

Major Challenges

• Management autonomy to the T1 and C1 groups

• Determination of the real populations sizes served by each health facility especially in urban settings

• Collaboration between PPA and the C2 health facilities o Little or no collaboration in preparation of business plans and signing of contracts (due to

lack of financial motivation)

o Not always ready to declare production for validation as this doe not lead to financial incentives

• Purchasing of services provided to clients from outside the catchment area. Case of referral hospitals serving other regions and other countries

• Initial level of development of the health facilities

From design to implement the

managerial autonomy intervention

in Benin Phase: Preparation and implementation

Maud Juquois

Context in Benin before RBF pilot

• Before RBF implementation (March 2012): o no autonomy for the HFs to spend their revenues (from user fees);

o The Health District Coordinator has to approve proposed

expenditures from user fees revenues (funds can stay few months in

the bank account if no supervision of the HD)

• From other countries RBF successes: o RBF can have a stronger impact if managers are more autonomous;

o Proposition for the IE RBF in Benin : to test this specific point

through a cross-cutting randomized trial.

Design de l’Impact Evaluation in Benin

• A combination of two interventions is tested:

o RBF “conditional rewards” (credits linked to results achieved by

health centers) versus “unconditional” rewards (credits not linked to

results achieved).

o Management autonomy versus no management autonomy.

• Therefore, there are currently five groups in the IE:

RBF treatment

(85 FS)

RBF control (additional

financing)

(87 FS)

No intervention

Management autonomy

treatment

(84 HF)

T1

40 HF

T2

44 HF

-

Management autonomy control

(88 HF)

T3

45HF

T4

43HF

-

No intervention - - C

46 HF

Implementation of this intervention

• With RBF : o One group of HFs with some increased managerial autonomy: information only to the

HD manager of RBF business plans;

o One group of HFs with no increased managerial autonomy: approval of the business

plan needed.

• But RBF funds can not be used to recruit health workers or

do some works;

• This intervention was seen as a 1st step to then further

improve autonomy of HF.

No real difference between the 2

groups (at primary health centers)

0

100

200

300

400

500

600

700

800

Number of ANC4

RBF+managerial autonomy

RBF only

0100200300400500600700800900

1000

Growth monitoring under 5

RBF+managerial autonomy

RBF only

Explanations • 1. Managerial autonomy in the control and

intervention facilities remains very limited, the

difference between the 2 arms is very thin;

• 2. Examination of how funds are spent at the

health facility level o Using part of RBF funds for operating costs is perceived as cumbersome

and requires more documentation than for revenues generated by user

fees;

o Staff also fear audits of RBF funds.

Conclusion • Difficult to implement an intevention arm as « managerial

autonomy », even at a limited stage, when not fully endorse

by the authorities at district and national levels;

• How to increase managerial autonomy from now in Benin? o Develop and disseminate the best examples of more autonomous HF with better

RBF results;

o Coaching of Health Facilities ;

o Training of districts coordinators in « results management » which could help

demonstrate benefits of an increased autonomy of HF to better perform.

o Better align control mechanisms over revenues generated by RBF and user fees

Case study from Karnataka, India Phase: Analysis and dissemination

Neeraj Sood

The Intervention: Vajpayee

Aarogyashri Scheme(VAS)

• Poor households are auto-enrolled in VAS o Receive free care at both public and private hospitals

o Goal is to reduce unmet need for expensive care and improve health outcomes

• VAS covers tertiary care for the following conditions o Cardiac care

o Cancer

o Neurological diseases

o Renal disease

o Neonatal problems/birth defects

o Burns

o Poly trauma

• Hospitals receive fixed reimbursement for predefined bundle of services (packages) o 402 tertiary care service packages

Empirical Strategy for Baseline Data

0 80 16040 Kilometers

VAS

Non-VAS

BellaryHaveriUttara

Kannada

Shimoga

Davangere

Chitradurga

Geographic regression discontinuity

• In 2010 VAS was arbitrarily

rolled out to only half the state

of Karnataka (northern part)

• Survey households close to the

north-south or eligibility border

• Used matching strategy to

further ensure similarity

between eligible and ineligible

areas

• Compare outcomes across

eligible and ineligible areas

VAS Eligible

VAS Ineligible

VAS Lowered Mortality for Covered Conditions

Poor Rich

Discussion • Implementation Issues

o Timing

o Retention

o Monitoring

• Innovation o Listing data

o Potential uses of baseline data

o Indirect tests of empirical strategy

• Dissemination challenges

Decision Maker Perspective on

Impact Evaluation • Key challenges in conducting impact evaluation

• Vision for using impact evaluation to improve decision

making

Stages of impact evaluation

Concept Note

Finalize Program and

IE Design

IE Baselin

e

Implement RBF

IE Endline

Analysis and Disseminati

on

Central African Republic

Cameroon

Benin

India

Discussion • What are the key lessons learned from these case studies?