effective vaccine management (evm) global data analysis€¦ · the effective vaccine management...

Effective Vaccine Management (EVM)

Global Data Analysis2010-2013

1

EVM – setting a standard for the vaccine supply chain

Update: WHO EVM database, December 2014

The Effective Vaccine Management (EVM) Assessmentg

2

A representative sample of sites is selected at each level of the supply chain.

Each of the 9 EVM Criteria is assessed at each supply chain level by observation, inspection of infrastructure and records, and by interview of staff.

Process and performance indicators are evaluated in each of the 9 areas at each level.

Indicator scores are combined to give criterion scores for each area at each level.

An area of vaccine management is considered “Effective” if its criterion score is greater than or equal to 80% - the EVM standard.

What does EVM assess?

How does EVM assess?

EVM assesses each level of the supply chain

Vaccine Manufacturer

or UNICEF

National store(Level PR)

Provincial stores(Level SN)

District stores(Level LD)

Health facilities(Level SP)

Primary level (PR):Vaccine stores that receive vaccine direct from an international vaccine manufacturer or distributors or a local vaccine manufacturer.

Sub-national level (SN):Vaccine stores that receive vaccine from a primary store or higher level sub-national store. There may be 0, 1, or more SN levels.

Lowest distribution level (LD):Vaccine stores that receive vaccine from a primary level or a sub-national store and supply vaccine to one or more health facilities.

Service point level (SP):Facilities that receive vaccine from any higher level store and supplyimmunization services.

E8Vaccine

management E7

Distribution

E3Capacity

E2Storage

temperature

E4Buildings,

equipment,transport

E6Stock

management

E5Maintenance

E1Pre-shipment

& arrival

E9Information

systems &supportive functions

EVM assesses 9 areas of vaccine management – the 9 EVM ”Criteria”

Assessments (2010-2013)

3

2010 (5 assessments)2011 (30)2012 (20)2013 (11)Data Source: WHO EVM database, November 2014

Map production: Immunization Vaccines and Biologicals, (IVB), World Health OrganizationDate of slide: 10 September 2014

The boundaries and names shown and the designations used on this map do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate border lines for which there may not yet be full agreement. WHO 2014. All rights reserved

75 countries have conducted at least one EVM assessment between 2010 and 2013. 4 countries, Senegal, Vietnam, Cameroon and Sierra Leone, have conducted 2 assessments. For those countries that have conducted 2 assessments, only the years of the most recent assessments are shown.

0 875 1,750 3,500 Kilometers

EVM Criterion Scores

2010-2013

44



EVM Criterion Scores (2010-2013)

5

E1 E2 E3 E4 E5 E6 E7 E8 E9

Primary (70 assessments)

Sub-national (53)

Lowest distribution (70)

Service point (70)

Globally, there is a broad range of performance in each criterion at each level; only E3 at the primary level has a median score above 80%. E3 Storage and transport capacity and E4 buildings and equipment are relatively strong at each level of the supply chain. Among the weakest areas are E2 Temperature monitoring at the national level, E5 Maintenance at the lower levels, E6 Stock management at

the lower levels, and E7 Distribution to the lower levels.

25th percentile

Maximum score

Median score

Minimum score

75th percentile

Vaccine arrival

Temperature monitoring

Storage capacity

Buildings & equipment

Maintenance Stock management

Vaccine distribution

Vaccine management

Information systems

0 1,700 3,400850 Kilometers

6

EVM Country Composite Scores (2010-2013)

Data Source: WHO EVM database. November 2014Map production: Immunization Vaccines and Biologicals, (IVB), World Health OrganizationDate of slide: 23 February 2015

The boundaries and names shown and the designations used on this map do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate border lines for which there may not yet be full agreement. WHO 2014. All rights reserved

Percentiles 26 to 50 (17 countries)

Percentiles 51 to 75 (17 countries)

Top 25% (18 countries)

Bottom 25% (18 countries) 0 875 1,750 3,500 Kilometers

EVM Criterion Scores by Region

2010-2013

77



E1 Vaccine arrivals (2010-2013)

8

25th percentile

Maximum score

Median score

Minimum score

75th percentile

AFR EMR EUR AMR SEAR WPR(n=37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6)

EMR and EUR score higher on average than the other regions. More than half of their primary stores assessed surpass the EVM standard. Less than 25% of the primary stores in the other regions meet the standard.

E2 Temperature monitoring (2010-2013)

9

Primary (PR)

Sub-national (SN)

Lowest distribution (LD)

Service point (SP)

25th percentile

Maximum score

Median score

Minimum score

75th percentile

More than 75% of countries in EMR & EUR meet the standard at LD & SP; more that 75% of EUR countries also meet the standard at SN. Otherwise, with very few exceptions, temperature monitoring is below the EVM standard of >= 80%. The PR E2 scores are low in all countries in all regions.

AFR EMR EUR AMR SEAR WPR(37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6)

E3 Storage capacity (2010-2013)

10

Primary

Sub-national

Lowest distribution

Service point

25th percentile

Maximum score

Median score

Minimum score

75th percentile

At each level of the supply chain in AFR, about 75% of the countries assessed have significant storage capacity shortfalls. The situation is slightly better in SEAR and WPR; at each level about 50% of countries have significant shortfalls. At each level in EMR and EUR, less than 25% of the countries have shortfalls.


E4 Infrastructure (2010-2013)

11

Primary

Sub-national

Lowest distribution

Service point

25th percentile

Maximum score

Median score

Minimum score

75th percentile

AFR has slightly poorer infrastructure overall than in other regions; at each level less than 25% of countries meet the EVM standard of >= 80%. More than 75% of primary stores assessed in the other regions meet the EVM standard. More than 75% of countries in EMR and EUR also meet the standard at SN level, and 3 of the 4 EMR countries meet the standard at LD.


E5 Maintenance (2010-2013)

12

Primary

Sub-national

Lowest distribution

Service point

25th percentile

Maximum score

Median score

Minimum score

75th percentile

Only the PR and SN median scores in EMR exceed the 80% EVM standard for preventive maintenance of equipment and buildings; all 4 PR stores and 3 of the 4 SN levels assessed meet the EVM standard.

The maintenance of cold chain infrastructure gets progressively weaker as vaccine flows through the supply chain from PR to SP.


E6 Stock management (2010-2013)

13

Primary

Sub-national

Lowest distribution

Service point

25th percentile

Maximum score

Median score

Minimum score

75th percentile

With a couple of exceptions, none of the regions meet the EVM standard of >= 80% in stock management. The exceptions are the PR, SN and LD levels in EMR; all 4 PR stores and 3 of the 4 SN and LD levels assessed score above 80%. Stock management procedures and performance gets progressively weaker as vaccines flow through the supply chain from PR to SP.


E7 Distribution (2010-2013)

14

Primary

Sub-national

Lowest distribution

Service point

25th percentile

Maximum score

Median score

Minimum score

75th percentile

In general, all regions perform poorly in the organization of vaccine distribution between the different levels of the supply chain. EMR and EUR countries score slightly higher in general than the countries of other regions.


E8 Vaccine management (2010-2013)

15

Primary

Sub-national

Lowest distribution

Service point

25th percentile

Maximum score

Median score

Minimum score

75th percentile

EMR and EUR score highly in vaccine management; at each level more than 75% of the countries meet the EVM standard of >= 80%. In AFR, PAH, SEAR and WPR, at each level of the supply chain, less than 25% of countries meet the EVM standard (exception WPR PR).


E9 Information systems (2010-2013)

16

Primary

Sub-national

Lowest distribution

Service point

25th percentile

Maximum score

Median score

Minimum score

75th percentile

With a few exceptions, information systems are weak at each level in each region. EMR PR and SN and EURO SN and LD are exceptions; more than 75% of countries assessed meet the EVM standard of >= 80% at these levels.


Selected EVM Indicator Scores

2010-2013

1717



Explanatory note:

There are 400 indicators at PR level, 300 at SN, 200 at LD and 150 at SP.

Most indicators assess compliance with EVM standards (Eg. % of cold rooms with continuous temperature monitoring). Some indicators measure performance (Eg. % of facilities with accurate stock records). Some indicators describe the context (Eg. % of stores that store vaccine at -20°C).

Such “context” indicators are not scored, but are used to customize the questionnaire – they turn OFF non-applicable indicators.

The following slides present global results for selected indicators. The first 3 slides show results for selected “context” indicators. The remaining slides present results for selected scoring indicators.

The scoring indicators are grouped into 3 categories: Availability:those indicators likely to have a direct impact on the availability of vaccine at the service delivery level. Quality: those indicators likely to have a direct impact on the quality (potency) of vaccine at the service delivery level. Efficiency: those indicators likely to have an impact on the operational cost of the immunization supply chain.

Indicator data exist for 58 countries across 6 WHO regions: 67 primary stores 406 sub-national stores 896 lowest distribution stores 949 immunization service facilities.

Selected EVM Indicator Scores (2010-2013)

18

Context 1: Infrastructure (2010-2013)

19

% of vaccine stores that store vaccine at -20°C

% of facilities that store vaccine in cold rooms or freezer rooms

% of refrigeration units that are CFC-free

% of facilities with voltage fluctuations greater than +/-15%

% of facilities that deliver or collect vaccine using refrigerated vehicles

PrimarySub-national

Lowest distributionService point

*The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions.

Context 2: Policy (2010-2013)

20

% of health facilities that have adopted the MDVP

% of health facilities that bury used and discarded vials on site

% of health facilities that incinerate safety boxes on site

PrimarySub-national



Context 3: Outsourcing (2010-2013)

21

% of national stores that outsource customs clearance

% of national stores that outsource transport services

% of national stores that outsource equipment maintenance services

PrimarySub-national



Availability indicators (2010-2013)

22

% of stock record templates that contain all required fields

% of facilities with matching vaccine and diluent stock levels (mismatch <1%)

% of facilities with accurate vaccine stock records

(mismatch with actual stock level <1%)

% of facilities that were able to fully satisfy all requests from lower level

facilities or for outreach in the past year

PrimarySub-national



Quality indicators (2010-2013), 1

23

% of countries that have carried out a systematic temperature monitoring

study within the past 5 years

% of cold rooms for which a fully documented temperature mapping

report is available

% of facilities in which all vaccine refrigerators comply with WHO

specifications

% of facilities for which kerosene or gas is always available

(where absorption refrigerators are used )

PrimarySub-national




24

PrimarySub-national



% of facilities in which all cold and freezer rooms have continuous temperature

recorders

% of facilities in which all vaccine refrigerators have continuous

temperature recorders or freeze indicators

% of facilities that pack freeze indicators with deliveries of freeze-sensitive

vaccines

% of health facilities in which all VVMs are before the discard point


25

PrimarySub-national



% of storekeepers and health workers that know which vaccines on the

schedule can be damaged by temperatures below 0°C

% of storekeepers and health workers that can condition icepacks or cool

chilled water packs in accordance with WHO guidelines

% of storekeepers and health workers that know when and how to conduct the

shake test

% of storekeepers and health workers that know how to read VVMs

Efficiency indicators (2010-2013), 1

26

PrimarySub-national



% of facilities with an up to date cold chain equipment inventory

% of refrigerators that are fully functional

% of facilities with a functional telephone, radio or internet connection

% of facilities with a computerised stock management system


27

PrimarySub-national



% of facilities that calculate vaccine wastage rates for each vaccine

% of facilities that use vaccine wastage rate data to forecast vaccine needs

% of facilities that lost less than 1% of vaccine stock in the past year due to

temperature damage

% of vaccine arrivals for which there is a correctly completed VAR


28

PrimarySub-national



% of facilities with a Standard Operating Procedures (SOP) manual

% of facilities with a satisfactory SOP describing a contingency plan in the event of equipment failure or other

emergency

% of facilities that receive regular supportive supervision

% of health workers that know how to apply the MDVP

Links

29

EVM public access website ( http://www.who.int/immunization/programmes_systems/supply_chain/evm/en/ )

A one-stop shop for all things EVM:

Download EVM tools and user guides:

Background EVM documents

EVM assessor training materials

EVM assessment report template

EVM site selection tool and user guide

EVM Assistant tool and user guide. This tool facilitates the calculation of available and required storage and transport capacities.

EVM Analysis & Recommendations tool. This tool facilitates the identification of gaps and drafting of recommendations.

Link to EVM Assessors website ( https://extranet.who.int/evm/ )

Download latest versions of the offline assessment tool and questionnaire.

Upload completed EVM assessment reports and datasets.

This website is for registered EVM assessors only. To register you must first complete a WHO EVM assessor training course.

Link to EVM e-learning website( http://apps.who.int/immunization_delivery/systems_policy/logistics/evmlearning/index_0_1_1.php )

An online training course on how to use the EVM assessment tools.

This course compliments the formal WHO training course, it does not replace it.

It is highly recommended that you complete this training BEFORE you start using the EVM tools.

http://www.who.int/immunization/programmes_systems/supply_chain/evm/en/

https://extranet.who.int/evm/

http://apps.who.int/immunization_delivery/systems_policy/logistics/evmlearning/index_0_1_1.php

Slide Notes

3030


Slide Notes

31

Slide 2

Each criterion is assessed at each level by observation, inspection and interview:

• Inspection of cold chain equipment, transport vehicles and buildings

• Inspection of records (temperature, stock, wastage, …) for the previous 12 months

• Interview of responsable staff to assess knowledge, understanding and practice

Note that E1 is assessed at the Primary level only, and E9 was not assessed at the Service Point level in the original version of the questionnaire (version 1.0), but is in the new questionnaire (version 2.1)

Indicators are evaluated in each of the 9 EVM criterias at each level.

• 400 indicators at PR level, 300 at SN, 200 at LD, 150 at SP

• “Critical” indicators are given a weight of 5, non-critical indicators are given a weight of 1

Each criterion at each level is scored out of 100%.

• Each criterion score is the weighted average of its constituent indicator scores

Slide 5

Countries are assessed in 9 areas of vaccine management, the 9 EVM criteria: E1 Vaccine arrivals, E2 Temperature monitoring, E3 Storage and transport capacity, E4 Buildings and equipment, E5 Maintenance, E6 Stock management, E7 Vaccine distribution, E8 Vaccine management, E9 Information systems. Note that E1 is assessed at the Primary level only

Criterion score data is available for 70 of the 75 countries that have conducted at least one assessment. There are 70 assessment scores in the PR, LD and SP distributions, and 53 in the SN distributions because 17 of the national supply chains assessed have only 3 levels.

The box and whisker plot shows the median scores, the inter-quartile range (the range of scores of the central half) and the maximum and minimum scores. Consider the E1 PR box and whisker for example: the median score is about 70%, the central half of countries have a score between about 60% and 80%, about a quarter of countries have scores above the target score of 80%, a quarter have scores below 60%. At least one country scored 100% (maximum), and the lowest score was about 36% (minimum).

Slide Notes

32

Slide 6

A country’s EVM composite score is defined as the geometric mean of the country’s 32 individual criterion scores (PR E1, PR E2, …, SP E9). The EVM composite score is used to represent the overall strength of a country’s immunization supply chain. In the understanding that an immunization supply chain is “only as strong as its weakest link”, the geometric mean is preferred to the arithmetic mean, as low criterion scores have a stronger effect on the geometric mean (weak links have a more significant effect on the strength of the overall chain).

This map presents the overall distribution of countries' EVM composite scores by quartiles. The bottom 25% are the poorest performing relative to other countries who have conducted EVM assessments over the period 2010-2013, while the top 25% can be considered those countries who have the highest performing immunisation supply chain systems relative to other countries.

If a country has completed more than one assessment, the score of the most recent assessment is used.

14 countries (Afghanistan, Armenia, Moldova, Rwanda, Albania, Burkina Faso, Malawi, Sri Lanka, Tanzania, Uzbekistan, Vietnam, Zimbabwe, Yemen, Sudan) have country composite scores above the target score of 80%.

Slide 7

In summary:

• EMR and EUR perform significantly better on average than the other regions in the largely procedural areas of vaccine arrivals (E1), temperature monitoring (E2), stock management (E6), vaccine management (E8) and information systems (E9).

• AFR has the most room for improvement in the capacity, quality and condition of vaccine storage buildings and refrigeration equipment (E3 & E4).

Slide 8

AFR = Africa Region, EMR = Easter Mediterranean Region, EUR = Europe Region, AMR = Americas Region, SEAR = South East Asia Region, WPR = Western Pacific Region.

The assessment of vaccine arrivals procedures applies to the primary level only.

Slide Notes

33

Slide 9

There are more stringent temperature monitoring requirements at the PR level than at the lower levels due to the higher volume of vaccine stored at that level. This partially explains the relatively low scores at the PR level compared to the other levels.

Slide 15

The SP level has stricter requirements in vaccine management than the other levels, given that vaccines are administered at this level.

Slide 16

7 of the 66 countries were assessed using the new questionnaire (version 2.1) , in which E9 is assessed at SP level (3 in AFR, 3 in EMR and 1 in PAH).

Slide 18

The data was collected between 2010 and 2013 in 58 countries in 6 WHO regions (31 AFR, 4 EMR, 10 EUR, 1 AMR, 7 SEAR, 5 WPR).

Slides 19-28

The data was collected between 2010 and 2013 in 58 countries in 6 WHO regions (31 AFR, 4 EMR, 10 EUR, 1 AMR, 7 SEAR, 5 WPR).

Results are based on 67 primary stores, 406 sub-national stores, 896 lowest distribution stores, and 949 immunization service facilities.

Slide 19

The infrastructure context questions reveal that more than 10% of national stores assessed do not have cold or freezer rooms, that more than 10% do not store any vaccine at sub-zero temperatures, and that refrigerated vehicles are used in only a quarter of countries.

Slide Notes

34

Slide 20

The policy context questions reveal that the multi dose vial policy has yet to be implemented in about 20% of health facilities assessed.

MDVP = Multi-dose vial policy.

Slide 21

The outsourcing context questions reveal that many countries outsource services such as customs clearance, vaccine transport and equipment maintenance.

Slide 22

The availability indicators reveal that more than half of the stock record templates in use do not contain all of the recommended fields, that more than a third of facilities have inaccurate stock records, have mismatched vaccine and diluent stocks, and have been unable to meet demand on at least one occasion in the year preceding assessment

Slide 23

The quality related indicators reveal poor temperature monitoring practice and equipment. Less than a fifth of countries have conducted a temperature monitoring study in the past 5 years, less than a fifth of cold rooms have been temperature mapped.

Slide 24

Less than a half of all cold rooms and refrigerators have the recommended temperature monitoring equipment, and monitoring of exposure to freezing temperatures during transport in passive containers with ice-packs is almost non-existent

VVM = Vaccine Vial Monitor

Slide Notes

35

Slide 25

On the bright side, the quality related questions do reveal a high level (>80%) of health worker and store manager knowledge and understanding of vaccine temperature sensitivities and recommended vaccine storage temperatures, and use of VVMs. The shake test is however not well understood, especially by health workers at the service delivery level (40%).

VVM = Vaccine Vial Monitor

Slide 26

The efficiency related questions reveal that about 50% of the countries assessed do not have an up to date cold chain equipment inventory. In addition, the data reveal that about 25% and 50% of national and sub-national stores respectively still use paper based stock management systems.

Slide 27

The efficiency related questions reveal that more than a third of vaccine storage facilities do not monitor vaccine wastage.

VAR = Vaccine arrival Report

Slide 28

The efficiency related questions reveal that less than 50% of national stores do not have a set of standard operating procedures, and that more than a third of health workers and store managers do not receive regular supportive supervision.

MDVP = Multi-dose vial policy

effective vaccine management (evm) global data analysis€¦ · the effective vaccine management...

Documents