assessment of malaria outcome indicators using lot quality assurance sampling (lqas): estimation of...
TRANSCRIPT
Assessment of Malaria Outcome Indicators Using Lot Quality Assurance Sampling (LQAS):
Estimation of Bed Net Coverage in Mozambique
Caitlin BiedronASPH M&E Fellow
CDC-RwandaJanuary 15, 2009
Lot Quality Assurance Sampling (LQAS)
• Classification method used in industrial sampling to identify batches of products (or lots) with an unacceptable number of defective items.
• Typically implemented as part of a stratified random sampling design: small samples are selected from all lots in a given area.
• In each lot, the sample determines whether coverage by a health intervention exceeds a specific target by using a statistically determined decision rule.
LQAS Terminology
• A lot typically consists of a supervision area, such as a health district.
• Lots classified as being acceptable/unacceptable vis-a-vis the target.
• In this analysis lots are enumeration areas (what-if analysis).
• The decision rule is the minimal number of individuals in the sample that should have the intervention.
• Selected to determine whether a population coverage target for an intervention has been reached, such as 80% vaccination coverage.
• If this coverage target is deemed to have been reached, the lot is classified as acceptable.
LQAS Methodology
The decision rule used when applying the LQAS method can be
represented with the composite hypothesis test:
The null hypothesis will be rejected if it is determined that the coverage proportion (p) of the lot is below an adequate level of coverage (pc).
The decision rule is defined by pc , the threshold value of coverage below which program managers deem services to be unacceptable.
:
:
O c
A c
H p p
H p p
LQAS Methodology
The α error is also commonly referred to as type I error, termed the provider risk in the LQAS context. A type I error is made if one rejects the null hypothesis when it is true. It is labeled as provider risk because the provider is at risk when the health area is identified as inadequately covered by an intervention when it has in fact met the target coverage level.
The classification of substandard lots as acceptable is called consumer risk, commonly referred to as the β error or type II error. A type II error is made when one fails to reject the null hypothesis when it is false.
The magnitude of the classification error varies both with the sample size and the maximum number of units without the intervention permitted in the sample (d).
Mozambique MIS
• Mozambique MIS was conducted by the NMCP in partnership with the Malaria Consortium and PMI.
• Included a sample drawn from a subset of EAs from a population proportionate sample from the 1997 census.
• A total of 346 EAs with 5990 households were selected from the 1510 EAs of the primary sample.
• After data cleaning, a total of 5745 household records were available for analysis.
• The survey was conducted June - July 2007.
LQAS Parameters
• We based our decision rule on a coverage target (pc) equal to 70% and defined our α error to be 10%.
• Year 1 Mozambique PMI target for bed net possession called for
70% of households to own at least one ITN.
• In urban clusters, the typical sample size was 20 households. In rural clusters, the typical sample size was 15 households.
• Based on these parameters, we calculated the decision rule for both urban and rural enumeration areas.
• The decision rule for urban clusters was 12 households; for rural clusters it was 8 households.
LQAS Results: Any Bed-net (Urban EAs)
TABLE 1. LQAS results for household possession of any bed-net in Manica, Mozambique, 2007
Decision Rules set for α<0.10
Cluster ID No Yes Total Decision Rule LQAS
175 11 8 19 11 failure
176 6 14 20 12 success
177 12 8 20 12 failure
178 8 12 20 12 success
179 8 11 19 11 success
180 10 10 20 12 failure
181 2 15 17 10 success
182 6 14 20 12 success
183 15 5 20 12 failure
184 8 12 20 12 success
185 7 14 21 12 success
186 11 9 20 12 failure
187 7 13 20 12 success
LQAS Results: Any Bed-net
TABLE 3. Coverage proportions, confidence intervals and LQAS result summaries for household possession of any bed-net in Mozambique
Province Coverage Proportion 95% Confidence Interval EAs w/adequate coverage
Niassa 0.406 (0.323, 0.489) 12 (34)
Cabo Delgado 0.376 (0.305, 0.448) 8 (34)
Nampula 0.300 (0.247, 0.353) 5 (36)
Zambezia 0.310 (0.243, 0.377) 6 (36)
Tete 0.318 (0.241, 0.394) 9 (34)
Manica 0.427 (0.363, 0.492) 12 (28)
Sofala 0.512 (0.445, 0.579) 18 (34)
Inhambane 0.315 (0.253, 0.377) 10 (34)
Gaza 0.368 (0.299, 0.437) 6 (24)
Maputo Province 0.335 (0.281, 0.389) 3 (32)
Maputo Cidade 0.481 (0.416, 0.546) 5 (20)
LQAS Results: Any ITN
TABLE 4. Coverage proportions, confidence intervals and LQAS result summaries for household possession of any ITN in Mozambique: 2007
Province Coverage Proportion 95% Confidence Interval EAs w/adequate coverage
Niassa 0.178 (0.130, 0.226) 0 (34)
Cabo Delgado 0.195 (0.134, 0.256) 3 (34)
Nampula 0.084 (0.053, 0.116) 0 (36)
Zambezia 0.143 (0.084, 0.201) 2 (36)
Tete 0.130 (0.081, 0.180) 1 (34)
Manica 0.368 (0.301, 0.435) 5 (28)
Sofala 0.228 (0.171, 0.284) 2 (34)
Inhambane 0.102 (0.072, 0.133) 0 (34)
Gaza 0.119 (0.072, 0.165) 0 (24)
Maputo Province 0.067 (0.042, 0.092) 0 (32)
Maputo Cidade 0.099 (0.068, 0.131) 0 (20)
LC-LQAS Background
• May not be feasible to conduct LQAS in the traditional fashion; instead only a subset of the areas in the catchment region represented would be selected.
• Pooled data would no longer be a stratified random sample, but instead a cluster sample, if the areas had been chosen at random.
• The recently developed large-country lot quality assurance sampling (LC-LQAS) method is used to integrate LQAS with cluster sampling.
• We next investigated the possibility of taking only a sub-sample of the EAs included in each province to calculate the provincial and national coverage estimates.
• This second aggregation was done in order to compare the estimates obtained using a smaller sample (220 EAs) to those estimates resulting from the full sample (346 EAs).
LC-LQAS Results: Any Bed-net
TABLE 5. LC-LQAS sub-sample coverage proportions, confidence intervals and LQAS result summaries for possession of any bed-net
Province Coverage Proportion 95% Confidence Interval EAs w/adequate coverage
Niassa 0.395 (0.272, 0.518) 5 (20)
Cabo Delgado 0.395 (0.289, 0.500) 4 (20)
Nampula 0.316 (0.243, 0.390) 1 (20)
Zambezia 0.318 (0.210, 0.426) 3 (20)
Tete 0.355 (0.224, 0.486) 6 (20)
Manica 0.447 (0.373, 0.520) 9 (20)
Sofala 0.479 (0.390, 0.568) 10 (20)
Inhambane 0.350 (0.241, 0.459) 7 (20)
Gaza 0.374 (0.295, 0.452) 6 (20)
Maputo Province 0.330 (0.265, 0.394) 2 (20)
Maputo Cidade 0.481 (0.416, 0.546) 5 (20)
LC-LQAS Results: Any ITN
TABLE 6. LC-LQAS sub-sample coverage proportions, confidence intervals and LQAS result summaries for household possession of any ITN in Mozambique: 2007
Province Coverage Proportion 95% Confidence Interval EAs w/adequate coverage
Niassa 0.128 (0.077, 0.180) 0 (20)
Cabo Delgado 0.222 (0.123, 0.321) 1 (20)
Nampula 0.067 (0.026, 0.107) 0 (20)
Zambezia 0.158 (0.051, 0.266) 1 (20)
Tete 0.149 (0.041, 0.257) 0 (20)
Manica 0.402 (0.324, 0.481) 4 (20)
Sofala 0.161 (0.108, 0.214) 2 (20)
Inhambane 0.103 (0.059, 0.146) 0 (20)
Gaza 0.135 (0.085, 0.186) 0 (20)
Maputo Province 0.067 (0.034, 0.100) 0 (20)
Maputo Cidade 0.099 (0.068, 0.131) 0 (20)
Comparison of Coverage Estimates
TABLE 7. Provincial and national coverage estimates for aggregate LC-LQAS, LQAS, and MIS cluster-samples: Mozambique 2007
Province Any Bed-Net in HH Any ITN in HH
LC-LQAS sub
samplefull LQAS
sampleMIS Cluster
estimateLC-LQAS sub
samplefull LQAS
sampleMIS Cluster
estimate
Niassa 0.395 0.406 0.422 0.128 0.178 0.177
Cabo Delgado 0.395 0.376 0.378 0.222 0.195 0.196
Nampula 0.316 0.300 0.329 0.067 0.084 0.087
Zambezia 0.318 0.310 0.365 0.158 0.140 0.178
Tete 0.355 0.318 0.317 0.149 0.130 0.119
Manica 0.447 0.427 0.448 0.402 0.368 0.369
Sofala 0.479 0.512 0.504 0.161 0.228 0.217
Inhambane 0.350 0.315 0.323 0.103 0.102 0.112
Gaza 0.374 0.368 0.373 0.135 0.119 0.133
Maputo Prov. 0.330 0.335 0.297 0.068 0.067 0.057
Maputo Cidade 0.481 0.481 0.486 0.099 0.099 0.102
National Coverage 0.365 0.355 0.375 0.143 0.145 0.158
Strengths of LQAS
• Allows for pass/fail classification based on various decision rules; • Dichotomous nature of the results can provide local supervisors with
a decisive judgment about action to be taken; • Sample sizes are typically smaller than those required to perform
other estimation analyses;• Inexpensive relative to traditional surveys;• Surveys can capture variability across local areas, which
encourages local level monitoring and accountability;• Surveys can be conducted more often, providing information more
frequently than HH surveys;• Sample can be treated as a stratified sample for further analysis;• With adequate sampling and appropriate weighting, can provide
estimates at aggregate levels (provincial, national);• To accommodate large countries, sampling areas may be chosen
randomly based on cluster sampling methodology (LC-LQAS).
Limitations of LQAS
• A high level of technical competence is required to achieve more than the basic pass/fail classification;
• Understanding of sampling frame requires knowledge of household selection, weighting and binomial probability;
• Perception that methodology is ‘simple’ belies technical complexity;• Decision rules may seem arbitrary if they are not based on explicit
program targets or certain thresholds related to transmission;• Sample sizes tend to be small, so estimates may have large
confidence intervals when sample combined;• Pass/fail outcomes do not give information on scale of change at the
local level, and it is not clear how dichotomous outcomes will be used to track progress in programs;
• Survey EAs must be selected in accordance with health districts or administrative units in order for results to have programmatic relevance.
Potential Applications in the Field
• The demonstrated methods could play a tangible role in the field where there is a growing interest in obtaining coverage estimates for small geographic units.
• LQAS could be used to assess EA data collected by standard MIS surveys to provide more frequent tracking of coverage indicators for districts country-wide or within key target areas.
• The standard MIS methodology requires large sample sizes to obtain reliable district level point estimates, making the survey both cumbersome and costly to conduct.
• LC-LQAS could provide local and national level estimates for coverage when it replaces the standard MIS sampling.
• LC-LQAS methodology would be applicable in those countries in which survey sampling units are selected in accordance with health districts or administrative units.
Potential Applications in the Field
• LQAS offers an alternative to malaria control program managers who are interested in tracking coverage at a local level to improve their service delivery strategies and to adjust priorities.
• As more countries aim to control malaria and scale-up
coverage to do so, it may become necessary to offer alternatives to the national level surveys that provide a single set of indicators for a country and do not present differences across local levels.
• National malaria programs are increasingly interested in
getting data from smaller geographic areas; LQAS may be a suitable method for such programs to use.
Acknowledgements
Amy Ratcliffe, CDC Malaria BranchAlbert Kilian, Malaria ConsortiumMarcello Pagano, Harvard School of Public HealthMarcia Castro, Harvard School of Public HealthBethany Hedt, Harvard School of Public HealthJoe Valadez, Liverpool School of Tropical MedicineJuliette Morgan, CDC-Mozambique