presentation on the literature review of interventions to improve health care provider performance

Contact for this presentation: Alexander K. Rowe, MD, MPH Malaria Branch, Division of Parasitic Diseases and Malaria, Center for Global Health Centers for Disease Control and Prevention Mailstop F22 4770 Buford Highway Atlanta, GA 30341 United States

Telephone: 1-770-488-3588 Fax: 1-770-488-7761 Email: [email protected]

Presentation on the literature review of interventions to improve health care provider performance

(for the First Global Symposium on Health Systems Research, Session on Evidence on Improving Health Service Delivery, November 18, 2010)

Early results of a systematic review of strategies to improve health care provider performance in low- and

middle-income countries

Alexander K. Rowe (Malaria Branch, Centers for Disease Control and Prevention [CDC])Samantha Y. Rowe (CDC)David H. Peters (Johns Hopkins University)Kathleen A. Holloway (World Health Organization)John Chalker (Management Sciences for Health)Dennis Ross-Degnan (Harvard Medical School)

Background

• Health care providers (HCPs) play key roles in improving quality and coverage of health interventions

• In low- and middle-income countries (LMICs), however, HCP performance often inadequate

• Most existing reviews* of strategies to improve performance are outdated or have important methodological limitations

* Two new reviews on community/lay health workers

Objectives

Conduct systematic review of strategies to improve HCP performance and related health outcomes in LMICs, produce:

1. Library & database of studies on improving HCP performance, for policy-makers and researchers

2. Detailed report and summary articles on effectiveness and costs of strategies

3. Evidence-based policy recommendations (i.e., guidance on implementing clinical guidelines in LMICs) and associated research agenda

Methods: inclusion and exclusion criteria

• Source of studies. Published and unpublished

• Timing. For electronic databases, studies by May 2006; for other methods (e.g., gray literature), no a priori end of search date (i.e., some studies after 2006)

• Setting. LMICs, as per World Bank


• Source of studies. Published and unpublished

• Timing. For electronic databases, studies by May 2006; for other methods (e.g., gray literature), no a priori end of search date (i.e., some studies after 2006)

• Setting. LMICs, as per World Bank

• Type of health condition. No exclusions; performance related to any health condition is acceptable

• HCPs. Facility- or community-based health workers, pharmacists, shopkeepers who sell medicines, and private sector health workers; only exclusions were household providers (e.g., patient’s family)

• Types of outcomes– Direct measures of HCP behavior (e.g., tasks related

to diagnosis, treatment, counseling)

– Patient outcomes (e.g., mortality, health care utilization)

– Other (e.g., economic outcomes, HCP knowledge)


• Types of outcomes– Direct measures of HCP behavior (e.g., tasks related

to diagnosis, treatment, counseling)

– Patient outcomes (e.g., mortality, health care utilization)

– Other (e.g., economic outcomes, HCP knowledge)

• Sample size. >20 observations per study group and time point

• Language of the study. No exclusions


Methods: classification of study design

• “Adequate” study designs for primary analysis:

– Pre-post with comparison (+/- randomization)

– Post-only with randomized comparison group

– Interrupted time series (>3 data points before and after intervention)

Methods: classification of study design

• “Adequate” study designs for primary analysis:

– Pre-post with comparison (+/- randomization)

– Post-only with randomized comparison group

– Interrupted time series (>3 data points before and after intervention)

• Despite interest in other study designs, due to very large number of reports, focus on studies with adequate designs

Methods: literature search strategy

• Searched: 15 electronic databases (e.g., MEDLINE, CINHL, EPOC specialized register, etc.)

• Bibliographies of 510 previous reviews & other articles

• Document inventories and websites of 29 organizations involved with HCP performance (e.g., MSH, BASICS, Core Project, DFID, PAHO, QAP, USAID, World Bank)

• INRUD Bibliography and WHO Rational Use of Drugs database

• Asked colleagues for references & unpublished studies

Methods: data abstraction

• Double, independent abstraction with 20-page abstraction form

• Data entered into Access database

• Queries to study investigators (clarifications, details on contextual factors, etc.)

• Effect size in terms of %-point change

• If outcome is a percentage, effect size =(%POST – %PRE)intervention – (%POST – %PRE)control

• Calculate effect size such that positive = success

• If >1 outcome, take median effect size (MES) of primary outcomes

Methods: analysis of effect size

Early, illustrative results

(might not represent final dataset)

• >105,000 citations screened

• After removing duplicates: 2,430 reports identified for abstraction (all study designs)

Results: Literature search

Early results: data abstraction

• From August 2007, studies with “adequate” designs prioritized to be abstracted first (851 reports on about 425 distinct studies)

• To date, 463 reports double-abstracted on 275 distinct studies

• Studies from >60 LMICs (50% low-income)

• Wide range of health conditions addressed by HCPs: nutrition; antenatal care; treatment of ARI, diarrhea, or malaria; family planning, etc.

Early results: study designs (N=275)

• “Adequate” designs (223/275, or 81%)− 145 randomized studies (54%)

• Other designs (52/275, or 19%), e.g., pre-post study without controls (not primary focus)

Early results: strategies tested (N=275)

• 390 intervention groups (i.e., strategies tested)

• 456 comparisons− 177 intervention vs. “no intervention” control− 249 intervention A vs. intervention B

• Most strategies had multiple components (e.g., training + supervision = 2 components)

• Median of 4 components/strategy (range: 1–17)

• Commonly tested strategies include: training, supervision or feedback, community activities, new guideline, printed materials, and job aids

Early results on effectiveness:

Analysis of 213 comparisons from 172 studies with adequate design

(all comparisons are: intervention vs. no intervention control)

(might not represent final dataset)

0

5

10

15

20

25

30

35

Magnitude of effect size (percentage-point change)

No.

of M

ES

-100 to -109

-0.1 to -9

0 to

9

10 to 19

-20 to -29

-30 to -39

-10 to -19

20 to 29

40 to 49

50 to 59

30 to 39

60 or

higher

Distribution of 213 MES from 172 studies with adequate design (all strategies)

Min = -105 Median = 9 Max = 130 IQR: 3–23

0

5

10

15

20

25

30

35


No.

of M

ES

-100 to -109

-0.1 to -9

0 to

9

10 to 19

-20 to -29

-30 to -39

-10 to -19

20 to 29

40 to 49

50 to 59

30 to 39

60 or

higher


50% are small (<10%-points)

or negative


0

5

10

15

20

25

30

35


No.

of M

ES

-100 to -109

-0.1 to -9

0 to

9

10 to 19

-20 to -29

-30 to -39

-10 to -19

20 to 29

40 to 49

50 to 59


30 to 39

60 or

higher


-110

-60

-10

40

90

140

16 315No. comparisons

Trai

ning

onl

y

Job

aids

/ prin

ted

mat

. for

HCP

onl

y

Trai

ning

+ jo

b ai

ds o

r pr

inte

d m

at. f

or H

CP

Median MES (%-pts) 10 7 2 13 13 6 10 17 10 17 18 79 14

Trai

n +

qual

mgm

t +jo

b ai

ds/p

rinte

d m

ater

ials

for H

CP

Com

mun

ity

activ

ities

+ o

ther

Non-

com

mun

ity

activ

ities

invo

lvin

g co

mm

oditi

es +

oth

er

Trai

ning

+ q

ualit

ym

anag

emen

t

Qual

ity m

gmt

tech

niqu

e on

ly

Med

ian

effe

ct s

ize

(%-p

oint

s)

75th percentile

median

25th percentile

-110

-60

-10

40

90

140

16 315No. comparisonsMedian MES (%-pts) 10 7 2 13 13 6 10 17

10 17 18 79 14

Med

ian

effe

ct s

ize

(%-p

oint

s)

75th percentile

median

25th percentile

• All median values small/moderate (2–17 %-points)

• Some groups with large variation (some effects large; potential for positive deviance analysis)

-110

-60

-10

40

90

140

Med

ian

effe

ct si

ze (%

-poi

nts)

1 2 3 4 5 >6

No. of components in the strategy

Does MES vary by the number of components in the strategy?

-110

-60

-10

40

90

140

Med

ian

effe

ct si

ze (%

-poi

nts)

1 2 3 4 5 >6

No. of components in the strategy

75th percentile

median

25th percentile

Does MES vary by the number of components in the strategy?

• Extreme heterogeneity among studies– Settings– Outcomes– Interventions (e.g., not all “training” is same)

• MES crude summary measure that can mask variation in effect (improved methods in development)

• Precision very difficult to assess

Limitations

1. Surprisingly large evidence base on effectiveness of strategies, with many strategies tested in many settings

2. Effect sizes vary substantially: half have small effects, but some have large effects

3. Analyses to identify factors associated with high effectiveness

Early conclusions (1)

4. Evidence base seems fragmented– Researchers not building on each others’ work– Varying methods makes it difficult to put pieces

together

5. Standardizing methods & outcomes would improve ability to summarize evidence base and develop effective and practical recommendations; perhaps coordinated research agenda is needed

6. Final results pending (end of 2012)

Early conclusions (2)

Acknowledgments

• Charity Akpala• Tashana Carty• Adrijana Corluka• Didi Cross• Bhavya Doshi• Onnalee Gomez• Meg Griffith• Karen Herman• Qing Li• Connie Liu• Earl Long• Eliza McLeod• Dawn Osterholt

• Gabriel Ponce-de-Léon• Nancy Pulsipher• Atiq Rahman• Nirali Shah• Banafsheh Siadat• Sanja Stanojevic• Savitha Subramanian• Jeff Willis• Kindra Willis• Shannon Wood• Karen Wosje• Alicia Wright• Chunying Xie

• Special thanks to investigators who responded to queries

• Funding: Bill and Melinda Gates Foundation, CDC, World Bank

Thanks for your attention!

“Results! Why, man, I have gotten a lot of results. I know several thousand things that won’t work.”

Thomas A. Edison (1847–1931)

presentation on the literature review of interventions to improve health care provider performance

Health & Medicine