community learning systems : increasing accountability for results from development assistance joy...

Community Learning Systems: Increasing Accountability for Results from Development Assistance

Joy P. MukaireLawrence L. Jackson

Annual Conference 2009Christian Connections for International Health

The Shift in AccountabilityChange to a transitional phase should be reflected in the shift from the traditional role of NGOs in health of service provision

From little accountability to anyone other than the individual NGO and their donors

To a change demanding that NGOs implement health care on behalf of government, working alongside MoH GOSS to support national frameworks, supporting MOH in health systems strengthening at all levels.

Struggles of M&E SystemsHarmonization of M&E approaches Having one National Health Strategic FrameworkHaving one National M&E frameworkHaving one National Coordinating body

Strengthening and expanding M&E Capacity at all levelsPromoting decentralization of M&EImproving human capacity for M&EAllocating adequate resources to M&E

Struggles of M&E SystemsUsing national data to prescribe solutions for local level problems/needsTreating every community/district the same even when priorities differ in natureNot knowing the magnitude of the problem being addressedMismatch between national priority and local needs Promoting utilization of program dataTo decide on prioritiesMake tactical changes to improve programsTo scale up proven project models

Management for ResultsThere is increasing pressure to manage programs with focus on achieving results that are visible.Monitoring and Evaluation has two purposes:Aid program managers to improve their programs by using dataSub-national level (Implementation Teams)National level (Policy and Sr. Management)Support a reporting function National stakeholdersInternationally [Donors, International Organizations (RBM)]

Macro vs Local M&E Outcome Data NATIONAL LEVELProgramDistrict 1ProgramDistrict 2Local managers need M&E data that show what is happening in the areas where they work

Efforts to Strengthen M&E

Rapid Assessment using Lot Quality Assurance Sampling Sampling method used in industry since the 1920sAdapted to public health settings in mid-1980sLocal managers at a sub-district level using small samples determine whether a performance target has been reachedYes or No Judgment

What is LQAS?

An analysis method that can: be used locally in supervision areasto identify priority areas: those not reaching an established performance benchmark for an indicatordata for local management decision making and for sharing information across supervision areasmeasure coverage at an aggregate level (e.g., program catchment area or district or nation)be used locally in supervision areassuitable for Reporting PurposesLQAS uses small samples (less than 20)

ABCDEFGAssume a program areathat has 7 supervision areas or municipalities

Each one is supervised byone person

Each one has between 25-35 promoters/wards/communities to supervise

ABCDEFGGoodBelow Averageor EstablishedBenchmark

GoodBelow Average or Established Benchmark

Identify the reasons for program problemsDevelop targeted solutionsMaintain the program at the current level Identify Supervisors and Health Workers that can help other HealthWorkers improve their performance

Sheet1

LQAS Table: Decision Rules for Sample Sizes of 12-30 and Coverage Benchmarks or Average Coverage of 10% to 95%

Sample SizesCoverage Benchmarks or Average Coverage

10%15%20%25%30%35%40%45%50%55%60%65%70%75%80%85%90%95%

12N/AN/A112234556778891011

13N/AN/A1123345667889101111

14N/AN/A11234456788910111112

15N/AN/A122345667891010111213

16N/AN/A122345678991011121314

17N/AN/A1223456789101112131415

18N/AN/A12235678910111112131416

19N/AN/A12345678910111213141516

20N/AN/A12345678911121314151617

21N/AN/A123456891011121314161718

22N/AN/A123457891012131415161819

23N/AN/A1234678101112131416171820

24N/AN/A1234679101113141516181921

25N/A12245689101213141617182021

26N/A12345689111214151618192122

27N/A123457810111314151718202123

28N/A123457810121315161819212224

29N/A123457910121315171820212325

30N/A123457911121416171920222426

For all coverage benchmarks (except where noted) LQAS is at least 92% sensitive and specific

N/A = Not Applicable -- Indicates that LQAS should not be used since coverage is too low for LQAS to detect.

Alpha and Beta Errors are > 10%

Alpha and Beta Errors are > 15%

Sheet2

Sheet3

These are the problem supervision areas. The Decision Rule is 12 women delivering with a Trained Provider

Nica-CARE

Delivery Attended by a Clinically Trained Provider, Benchmark = 75%, DR=12; Mean = 65%, DR=10Delivery Attended by a Clinically Trained Provider Benchmark = 75%, Decision Rule=12; Mean = 65%, Decision Rule=10

NGO =CARE* = Below Benchmark; # = Below Mean

AREA | YES NO | TotalDistrict

--------------------------------------------AreaYesNoTotal

1 | 11 8 | 19111*819

2 | 13 6 | 19213619

3 | 19 0 | 19319019

4 | 8 11 | 1948* #1119

5 | 8 11 | 1958* #1119

6 | 15 4 | 19615419

--------------------------------------------Total7440114

Total| 74 40 | 114Mean Coverage in PVO Area = 65%

Nica-All-Delivery

Delivery Attended by a Clinically Trained Provider, Mean = 70.57%, DR=12

NGO =CARENGO =HOPE

AREA | YES NO | TotalAREA | YES NO | Total

----------------------------------------------------------------------------------------

1 | 11 8 | 191 | 11 8 | 19

2 | 13 6 | 192 | 13 6 | 19

3 | 19 0 | 193 | 4 15 | 19

4 | 8 11 | 194 | 7 12 | 19

5 | 8 11 | 19

6 | 15 4 | 19

----------------------------------------------------------------------------------------

Total| 74 40 | 114Total| 35 41 | 76

NGO =PARTNERSNGO =PCI


----------------------------------------------------------------------------------------

1 | 11 8 | 191 | 10 9 | 19

2 | 11 8 | 192 | 17 2 | 19

3 | 16 3 | 193 | 16 3 | 19

4 | 16 3 | 194 | 7 9 | 16

----------------------------------------------------------------------------------------

Total| 54 22 | 76Total| 50 23 | 73

NGO =PLANNGO =SAVE


----------------------------------------------------------------------------------------

1 | 19 0 | 191 | 17 2 | 19

2 | 13 6 | 192 | 14 5 | 19

3 | 15 4 | 193 | 16 3 | 19

4 | 15 4 | 194 | 16 3 | 19

5 | 16 3 | 195 | 14 5 | 19

6 | 15 4 | 196 | 14 5 | 19

7 | 19 0 | 19

----------------------------------------------------------------------------------------

Total| 93 21 | 114Total| 110 23 | 133

Nica-All-ORT

Demonstrates Correct Preparation of ORS Solution Mean = 77.89, DR = 13

NGO =ADRANGO =CARE


----------------------------------------------------------------------------------------

1 | 14 5 | 191 | 14 3 | 17

2 | 4 15 | 192 | 15 4 | 19

3 | 9 10 | 193 | 14 3 | 17

4 | 12 7 | 194 | 15 2 | 17

5 | 15 3 | 18

6 | 13 5 | 18

----------------------------------------------------------------------------------------

Total| 39 37 | 76Total| 86 20 | 106

NGO =CRSNGO =HOPE


----------------------------------------------------------------------------------------

1 | 16 3 | 191 | 15 4 | 19

2 | 19 0 | 192 | 15 4 | 19

3 | 17 2 | 193 | 18 1 | 19

4 | 17 2 | 194 | 17 2 | 19

----------------------------------------------------------------------------------------

Total| 69 7 | 76Total| 65 11 | 76

NGO =PARTNERSNGO =PLAN


----------------------------------------------------------------------------------------

1 | 14 5 | 191 | 15 1 | 16

2 | 13 6 | 192 | 18 1 | 19

3 | 13 6 | 193 | 18 1 | 19

4 | 17 2 | 194 | 15 2 | 17

5 | 17 2 | 19

6 | 14 2 | 16

----------------------------------------------------------------------------------------

Total| 57 19 | 76Total| 97 9 | 106

NGO =SAVE

AREA | YES NO | Total

--------------------------------------------

1 | 15 4 | 19

2 | 17 2 | 19

3 | 10 9 | 19

4 | 12 7 | 19

5 | 14 4 | 18

6 | 11 7 | 18

7 | 14 5 | 19

--------------------------------------------

Total| 93 38 | 131

Vitamin A

Vitamin A Received by Mothers within Two Months of Birth

Average = 40.8% DR=6

NGO =ADRANGO =CARE

VITAMINAVITAMINA


-----------+-------------+-----------------+-------------+------66

1 | 6 13 | 191 | 5 13 | 18212

2 | 2 17 | 192 | 11 8 | 1977

3 | 7 12 | 193 | 5 14 | 1966

4 | 6 13 | 194 | 4 15 | 19512

5 | 7 12 | 191111

6 | 10 9 | 19512

-----------+-------------+-----------------+-------------+------412

Total | 21 55 | 76Total | 42 71 | 11377

1010

NGO =PARTNERS1313

VITAMINA1515

AREA | YES NO | Total1616

-----------+-------------+------1919

1 | 13 6 | 19265126158

2 | 15 4 | 190.47547169810.5962264151

3 | 16 3 | 19

4 | 19 0 | 19

-----------+-------------+------

Total | 63 13 | 76

Nica-All-ITN

NicaSalud Baseline December 1999

Household has at least 1 BEDNET (Average= 39%, DR=5)

NGO =ADRANGO =CARE

BEDNETBEDNET


-----------+-------------+-----------------+-------------+------

1 | 1 18 | 191 | 9 10 | 19

2 | 4 15 | 192 | 3 16 | 19

3 | 8 11 | 19

4 | 10 9 | 19

5 | 4 15 | 19

6 | 3 16 | 19

-----------+-------------+-----------------+-------------+------

Total | 5 33 | 38Total | 37 77 | 114

NGO =CRSNGO =HOPE

BEDNETBEDNET


-----------+-------------+-----------------+-------------+------

1 | 6 13 | 191 | 4 15 | 19

2 | 3 16 | 192 | 5 14 | 19

3 | 4 15 | 19

-----------+-------------+-----------------+-------------+------

Total | 9 29 | 38Total | 13 44 | 57


BEDNETBEDNET


-----------+-------------+-----------------+-------------+------

1 | 4 15 | 191 | 10 9 | 19

2 | 4 15 | 192 | 6 13 | 19

3 | 3 16 | 19

-----------+-------------+-----------------+-------------+------

Total | 8 30 | 38Total | 19 38 | 57

NGO =PLANNGO =SAVE

BEDNETBEDNET


-----------+-------------+-----------------+-------------+------

1 | 4 15 | 191 | 7 12 | 19

2 | 17 2 | 192 | 10 8 | 18

3 | 10 9 | 193 | 10 9 | 19

4 | 16 3 | 194 | 12 7 | 19

5 | 15 4 | 19

6 | 15 4 | 19

-----------+-------------+-----------------+-------------+------

Total | 77 37 | 114Total | 39 36 | 75

Umoyo-All-HIV

Women Who Know HIV/AIDS Can

Be Transmitted During Pregnancy

Coverage Estimate=55.66%, DR=9

NGO = AHSNGO = BCC

AREA1.02.0TotalAREA1.02.0Total

181119161319

281119281119

361319351419

491019410919

571118581119

Total385694Total375895

NGO = EkwendeniNGO = MACRO


117219181119

216319291019

312719371219

412719411819

591019561319

613619612719


NGO = Malamulo

AREA1.02.0Total

116319Coverage Estimate=55.66%

215419Decision Rule=9

314519

418119

515419

Total781795

Regular Superv

1111

0.0230.0005290.00000027980

0.9190.8445610.71328328270.5087730414

0.8350.6972250.48612270060.2363152801

0.7370.5431690.29503256260.087044213

0.6330.4006890.16055167470.0257768403

0.5280.2787840.07772051870.006040479

0.4280.1831840.03355637790.0011260305

0.3370.1135690.01289791780.0001663563

0.2560.0655360.00429496730.0000184467

0.1880.0353440.00124919830.0000015605

0.1310.0171610.00029449990.0000000867

0.0870.0075690.00005728980.0000000033

0.0540.0029160.00000850310.0000000001

0.0310.0009610.00000092350

Session 1

Session 2

Session 3

Session 4

Proportion of Time Service is Done Correctly

ERROR

Regular Supervision of a Health Workers Observing them with 6 Clients and Permitting 1 Error in Service Provision: Service Standard = 95%

Sheet6

Error

QualitySession 1Session 2Session 3Session 4

100%

95%0.0230.0005290.00000027980

90.0%0.9190.8445610.71328328270.5087730414

85.0%0.8350.6972250.48612270060.2363152801

80.0%0.7370.5431690.29503256260.087044213

75.0%0.6330.4006890.16055167470.0257768403

70.0%0.5280.2787840.07772051870.006040479

65.0%0.4280.1831840.03355637790.0011260305

60.0%0.3370.1135690.01289791780.0001663563

55.0%0.2560.0655360.00429496730.0000184467

50.0%0.1880.0353440.00124919830.0000015605

45.0%0.1310.0171610.00029449990.0000000867

40.0%0.0870.0075690.00005728980.0000000033

35.0%0.0540.0029160.00000850310.0000000001

30.0%0.0310.0009610.00000092350

25.0%

20.0%

15.0%

10.0%

MBD001E3EA3.xls

Nica-CARE

Table 1: Delivery Attended by a Clinically Trained Provider Benchmark = 75%, Decision Rule=12; Mean = 65%, Decision Rule=10

PVO 1


--------------------------------------------

1 | 11 8 | 19

2 | 13 6 | 19

3 | 19 0 | 19

4 | 8 11 | 19

5 | 8 11 | 19

6 | 15 4 | 19

--------------------------------------------

Total| 74 40 | 114

Nica-All-Delivery

Delivery Attended by a Clinically Trained Provider, Mean = 70.57%, DR=12

NGO =CARENGO =HOPE


----------------------------------------------------------------------------------------

1 | 11 8 | 191 | 11 8 | 19

2 | 13 6 | 192 | 13 6 | 19

3 | 19 0 | 193 | 4 15 | 19

4 | 8 11 | 194 | 7 12 | 19

5 | 8 11 | 19

6 | 15 4 | 19

----------------------------------------------------------------------------------------

Total| 74 40 | 114Total| 35 41 | 76



----------------------------------------------------------------------------------------

1 | 11 8 | 191 | 10 9 | 19

2 | 11 8 | 192 | 17 2 | 19

3 | 16 3 | 193 | 16 3 | 19

4 | 16 3 | 194 | 7 9 | 16

----------------------------------------------------------------------------------------

Total| 54 22 | 76Total| 50 23 | 73

NGO =PLANNGO =SAVE


----------------------------------------------------------------------------------------

1 | 19 0 | 191 | 17 2 | 19

2 | 13 6 | 192 | 14 5 | 19

3 | 15 4 | 193 | 16 3 | 19

4 | 15 4 | 194 | 16 3 | 19

5 | 16 3 | 195 | 14 5 | 19

6 | 15 4 | 196 | 14 5 | 19

7 | 19 0 | 19

----------------------------------------------------------------------------------------

Total| 93 21 | 114Total| 110 23 | 133

Nica-All-ORT

Demonstrates Correct Preparation of ORS Solution Mean = 77.89, DR = 13

NGO =ADRANGO =CARE


----------------------------------------------------------------------------------------

1 | 14 5 | 191 | 14 3 | 17

2 | 4 15 | 192 | 15 4 | 19

3 | 9 10 | 193 | 14 3 | 17

4 | 12 7 | 194 | 15 2 | 17

5 | 15 3 | 18

6 | 13 5 | 18

----------------------------------------------------------------------------------------

Total| 39 37 | 76Total| 86 20 | 106

NGO =CRSNGO =HOPE


----------------------------------------------------------------------------------------

1 | 16 3 | 191 | 15 4 | 19

2 | 19 0 | 192 | 15 4 | 19

3 | 17 2 | 193 | 18 1 | 19

4 | 17 2 | 194 | 17 2 | 19

----------------------------------------------------------------------------------------

Total| 69 7 | 76Total| 65 11 | 76

NGO =PARTNERSNGO =PLAN


----------------------------------------------------------------------------------------

1 | 14 5 | 191 | 15 1 | 16

2 | 13 6 | 192 | 18 1 | 19

3 | 13 6 | 193 | 18 1 | 19

4 | 17 2 | 194 | 15 2 | 17

5 | 17 2 | 19

6 | 14 2 | 16

----------------------------------------------------------------------------------------

Total| 57 19 | 76Total| 97 9 | 106

NGO =SAVE


--------------------------------------------

1 | 15 4 | 19

2 | 17 2 | 19

3 | 10 9 | 19

4 | 12 7 | 19

5 | 14 4 | 18

6 | 11 7 | 18

7 | 14 5 | 19

--------------------------------------------

Total| 93 38 | 131

Vitamin A

Vitamin A Received by Mothers within Two Months of Birth

Average = 40.8% DR=6

NGO =ADRANGO =CARE

VITAMINAVITAMINA


-----------+-------------+-----------------+-------------+------66

1 | 6 13 | 191 | 5 13 | 18212

2 | 2 17 | 192 | 11 8 | 1977

3 | 7 12 | 193 | 5 14 | 1966

4 | 6 13 | 194 | 4 15 | 19512

5 | 7 12 | 191111

6 | 10 9 | 19512

-----------+-------------+-----------------+-------------+------412

Total | 21 55 | 76Total | 42 71 | 11377

1010

NGO =PARTNERS1313

VITAMINA1515

AREA | YES NO | Total1616

-----------+-------------+------1919

1 | 13 6 | 19265126158

2 | 15 4 | 190.47547169810.5962264151

3 | 16 3 | 19

4 | 19 0 | 19

-----------+-------------+------

Total | 63 13 | 76

Nica-All-ITN

NicaSalud Baseline December 1999

Household has at least 1 BEDNET (Average= 39%, DR=5)

NGO =ADRANGO =CARE

BEDNETBEDNET


-----------+-------------+-----------------+-------------+------

1 | 1 18 | 191 | 9 10 | 19

2 | 4 15 | 192 | 3 16 | 19

3 | 8 11 | 19

4 | 10 9 | 19

5 | 4 15 | 19

6 | 3 16 | 19

-----------+-------------+-----------------+-------------+------

Total | 5 33 | 38Total | 37 77 | 114

NGO =CRSNGO =HOPE

BEDNETBEDNET


-----------+-------------+-----------------+-------------+------

1 | 6 13 | 191 | 4 15 | 19

2 | 3 16 | 192 | 5 14 | 19

3 | 4 15 | 19

-----------+-------------+-----------------+-------------+------

Total | 9 29 | 38Total | 13 44 | 57


BEDNETBEDNET


-----------+-------------+-----------------+-------------+------

1 | 4 15 | 191 | 10 9 | 19

2 | 4 15 | 192 | 6 13 | 19

3 | 3 16 | 19

-----------+-------------+-----------------+-------------+------

Total | 8 30 | 38Total | 19 38 | 57

NGO =PLANNGO =SAVE

BEDNETBEDNET


-----------+-------------+-----------------+-------------+------

1 | 4 15 | 191 | 7 12 | 19

2 | 17 2 | 192 | 10 8 | 18

3 | 10 9 | 193 | 10 9 | 19

4 | 16 3 | 194 | 12 7 | 19

5 | 15 4 | 19

6 | 15 4 | 19

-----------+-------------+-----------------+-------------+------

Total | 77 37 | 114Total | 39 36 | 75

Umoyo-All-HIV

Women Who Know HIV/AIDS Can

Be Transmitted During Pregnancy

Coverage Estimate=55.66%, DR=9

NGO = AHSNGO = BCC


181119161319

281119281119

361319351419

491019410919

571118581119


NGO = EkwendeniNGO = MACRO


117219181119

216319291019

312719371219

412719411819

591019561319

613619612719


NGO = Malamulo

AREA1.02.0Total

116319Coverage Estimate=55.66%

215419Decision Rule=9

314519

418119

515419

Total781795

Regular Superv

1111

0.0230.0005290.00000027980

0.9190.8445610.71328328270.5087730414

0.8350.6972250.48612270060.2363152801

0.7370.5431690.29503256260.087044213

0.6330.4006890.16055167470.0257768403

0.5280.2787840.07772051870.006040479

0.4280.1831840.03355637790.0011260305

0.3370.1135690.01289791780.0001663563

0.2560.0655360.00429496730.0000184467

0.1880.0353440.00124919830.0000015605

0.1310.0171610.00029449990.0000000867

0.0870.0075690.00005728980.0000000033

0.0540.0029160.00000850310.0000000001

0.0310.0009610.00000092350

Session 1

Session 2

Session 3

Session 4

Proportion of Time Service is Done Correctly

ERROR

Regular Supervision of a Health Workers Observing them with 6 Clients and Permitting 1 Error in Service Provision: Service Standard = 95%

Sheet6

Error

QualitySession 1Session 2Session 3Session 4

100%

95%0.0230.0005290.00000027980

90.0%0.9190.8445610.71328328270.5087730414

85.0%0.8350.6972250.48612270060.2363152801

80.0%0.7370.5431690.29503256260.087044213

75.0%0.6330.4006890.16055167470.0257768403

70.0%0.5280.2787840.07772051870.006040479

65.0%0.4280.1831840.03355637790.0011260305

60.0%0.3370.1135690.01289791780.0001663563

55.0%0.2560.0655360.00429496730.0000184467

50.0%0.1880.0353440.00124919830.0000015605

45.0%0.1310.0171610.00029449990.0000000867

40.0%0.0870.0075690.00005728980.0000000033

35.0%0.0540.0029160.00000850310.0000000001

30.0%0.0310.0009610.00000092350

25.0%

20.0%

15.0%

10.0%

DistrictS.A. = 19S.A. = 19S.A. = 19S.A. = 19DistrictS.A. = 19S.A. = 19S.A. = 19S.A. = 19DistrictS.A. = 19S.A. = 19S.A. = 19S.A. = 19Nation or RegionSA= Supervision Area

LQAS Application in CHASApplied in two counties of Leer and Koch in Unity State to establish baseline for a CHAS implemented Access to Basic Service Quick Impact Project Estimate immunization coverage among children aged 12 59 months;Assess knowledge and practice of correct feeding practices among pregnant women and children aged 0 23 months;Estimate coverage of ITNs among children under 5 and pregnant mothers;Estimate prevalence of ARI, fever, and diarrhea among children under 5;

LQAS Application in CHASAssess knowledge and practice of mothers and caretakers of children under 5 of critical danger signs of and correct response to ARI, fever and diarrhea;Utilization of ANC, delivery and PNC services among pregnant women; and Knowledge and use of family planning methods among women in the reproductive age bracket.Use of sanitary means of excreta disposalUse of improved water source of drinking water

LQAS Application in CHASThe sample: A total of 5 parallel samples corresponding to the 5 project target groups, namely,Heads of householdMothers of children 0 11 monthsMothers of children 12 23 monthsMothers of children 0 59 monthsWomen aged 15 49From each randomly selected village133 households in Koch x 5 interviews = 152 households in Leer x 5 interview =

LQAS A Catalyst for Change

A team of 14 people were trained in Unity State, six at HQ and 8 at county levelMobilizes managers and their organizations to appraise their programsEmpowers them by creating a learning environmentSupports leadership at national, district and sub-district levels

Advantages of Using LQAS

Can be used at a local level with modest amounts of supervisionProduces information that can be rapidly interpreted by local managersIdentifies where the successes and challenges are locatedPaper/pencil analyses rather than requiring computer analyses Data can be used for national reporting, andSampling theory is rigorous

Lessons Learned

LQAS findings revealed substantial variations in estimates of countiesThe magnitude of Sanitation problem became more visible to implementers than before something the project was not even planning to address. The substantial variation imply that local managers are better placed to make tactical changes in their ProgrammeThose responsible for providing services are better placed to analyze the challenges in their areas and decide on the strategic change

The Christian Perspective

This is one solutions to increased STEWARDSHIP of faith-based organizations

This tool is just but a means to acceptable standards of ACCOUNTABILITY where resources are limited and competing priorities are many

This tool builds a culture of result-oriented management and the culture of accountability, a contribution Christian health services can make to Strengthening Health Information Systems and Monitoring and Evaluation

The Power of Measuring Results

If you dont measure results you cant tell success from failureIf you dont see success, you cannot reward itIf you cannot reward success, you are probably awarding failureIf you cannot recognize failure, you cannot correct itIf you cannot demonstrate results, you cannot win public supportSource: Adapted from Osborne & Gaebler 1992

Getting Ready

Household Interviews

Household Interviews

*************************

community learning systems : increasing accountability for results from development assistance joy...

Documents

national priority

national frameworks

systemsusing national

health systems

aggregate level

current level

program areathat

health care