12/04/2014

STRENGTHENING INDEPENDENT

IMMUNIZATION AND VACCINE

ADVISORY COMMITTEES (NITAG)

TECHNICAL CAPACITIES

Report of the East and Southern

African NITAG training workshop,

Naivasha (Kenya) 3-7 August 2014

2

Table of contents 1 Background 3

2 Objectives of the workshop 4

3 Method of work 4

3.1 Format 4

3.2 Topics on the agenda 4

4 Introduction 5

5 Training roll-out 5

5.1 Session1: roles and mode of operations of NITAGs 7

5.2 Session 2: Guidelines and tools for NITAGs functioning 9

5.3 Session 3: Monitoring the committee’s progress and achievements 10

5.4 Session 4: Development of an evidence-based recommendation- 1) defining

the recommendation framework 11

5.5 Session 5: Development of an evidence-based recommendation - 2) how to

search evidence 12

5.6 Session 6: Development of an evidence-based recommendation – 3) how to

evaluate data relevance and quality 13

5.7 Session 7: Development of an evidence-based recommendation – 4) how to

structure a recommendation note 14

5.8 Session 8: Communicating NITAG recommendations 15

5.9 Session 9: Networking 16

6 Conclusions 16

6.1 Next steps 16

6.2 Overall evaluation of the training 17

6.3 Considerations for future trainings 18

3

1 Background

Following on the WHO and its Strategic Advisory Group of Experts (SAGE) recommendations to

countries to establish National Immunization Technical Advisory Groups (NITAGs), the World

Health Assembly and other international immunization stakeholders have reiterated the

importance of establishing NITAGs that would be responsible for advising policy-makers on all

immunization-related aspects, including new vaccine adoption, strategies to increase use of

under-utilized vaccines and routine immunization programme performance, and evaluating

effectiveness of current immunization schedules. Establishment of NITAGs will ensure national

immunization policy decision process is more transparent, evidence-based, taking into account

local context therefore reinforcing the credibility of immunization policy, its sustainability and its

acceptance by the population.

The Supporting Independent Immunization and Vaccine Advisory Committees (SIVAC) Initiative

supports the creation and strengthening of sustainable NITAGs. The SIVAC Initiative is

implemented by the Health Policy and Institutional Development Unit of the Agence de

Médecine Préventive (AMP), a WHO collaborating Centre for the use of evidence-informed

immunization policy-making. The core of the initiative’s business is to build capacity at the

national level to establish and run a NITAG in accordance with WHO guidelines.

Between 2008 and 2013 the SIVAC Initiative has supported the creation or strengthening of 13

NITAGs and organized over 20 training workshops in collaboration with partners such as the

World Health Organization (HQ; Regional Offices; Evidence-Informed Policy Network (EVIPNet)

project), the International Vaccine Institute (IVI), US Centers for Disease Control (US-CDC), and

other units within AMP.

In East and Southern Africa, the SIVAC Initiative has supported Kenya and Uganda in the set-up

of their NITAG and has initiated collaboration the Republic of South Sudan and Zambia who

have expressed interest in setting-up/strengthening their NITAG. The initiative has also started

engaging Ethiopia in the discussion of the possible role of a NITAG in the country.

The SIVAC Initiative collaborated with WHO and CDC-Kenya to organize on August 4-7, 2014 in

Naivasha (Kenya) a regional training for Eastern and Southern African NITAGs. The workshop

brought together members of the Kenya, Uganda and South Sudan NITAGs, secretariats of

future NITAGs in Ethiopia and Zambia, and EPI focal persons in WHO and UNICEF country

offices. A total of 35 people participated in the workshop. The list of attendance is in annex 1 of

this report.

4

2 Objectives of the workshop

++++ To orient participants on NITAGs functioning: roles, responsibilities, mode of operations and evaluation

++++ To familiarize participants on the principles and methodology for issuing evidence-based recommendations, including tools adapted from EVIPNet, using case studies on flu vaccine and rotavirus vaccine

++++ To initiate mechanism for regional collaboration and experience sharing between NITAGs

3 Method of work

3.1 Format

The 3 ½ day interactive training workshop combined plenary sessions alternating with group

work. Reference materials (templates, check lists, articles and reports) were provided to facilitate

the group work. The training started with a pre-workshop assessment of participants’ skills in

developing evidence based recommendations and use of evidence search strategies. An

evaluation was conducted at the end of each day: participants were asked to rate each session

(using the scale : below average, average or above average) in terms of its comprehensiveness,

clarity and interest. A final evaluation was also conducted at the end of the training.

3.2 Topics on the agenda

The modules were selected from the 3 training units of the SIVAC Initiative training catalogue

and are listed below:

1. Training 1: Establishment and mode of operations of an effective NITAG

• Module 1: Role, scope and functions of a NITAG;

• Module 2: NITAG processes (tools);

• Module 3 : Communication of recommendations and involvement of stakeholders;

• Module 4: Evaluation of NITAG functionality, performance and impact.

2. Training 2: Analysis of Health Systems, Immunization and Policy-decision process

5

• Module 1: Landscape analysis of the health system and immunization context and

how it can impact proposed recommendation

3. Training 3: Technical and scientific capacities of NITAGs

• Module 1: Evidence assessment methodologies;

• Module 2: Development of an evidence-informed recommendation

The topics were covered in 8 sessions. In addition, a session was devoted to discussion around

networking and exchanges between NITAGs. The agenda of the workshop is shown in annex 2.

4 Introduction to the workshop

In his welcome note, the head of Kenya Unit of Vaccine and Immunization Services shared

Kenya experience in the process of establishing the advisory group ( the KENITAG) highlighting

some of the challenges faced along the way.

The Chair of the Kenya NITAG provided a brief introduction and the rationale for establishing

NITAGs; he presented the proposed structure and functioning of the KENITAG. The KENITAG

will be supported by a secretariat (UVIS); within the committee focal points are designated for

each of the following areas: VPD epidemiology, health systems and immunization, vaccinology,

monitoring and evaluation, policy.

The key note delivered by WHO Inter-country Support team for East and Southern Africa

highlighted the role NITAG may have in supporting the implementation of the Global Vaccine

Action Plan.

5 Training roll-out

The quick pre-training assessment of participants’ skills and knowledge in issuing evidence-based recommendation showed that the majority of them do not have experience in using a systematic process to identify and evaluate the quality of the evidence. While most of them have used scientific databases, the majority have not used information search strategies.

6

7

5.1 Session1: roles and mode of operations of NITAGs

The session started with a group exercise: participants were asked to list on a flip chart key

features they attribute to the various immunization committees existing in their countries in terms

of membership, structure, scope and mandate.

8

The aim of the exercise was to assess participants understanding of the differences between

NITAGs and other immunization committees.

Each group reported back to plenary. Bridging from the exercise, a presentation was made to

further clarify NITAGs role and mandate as recommended by the WHO. Participants from Kenya

and Uganda then shared the terms of reference of their respective NITAG, highlighting how

these compare to the WHO guidelines.

Take-away messages from the

session

Participants evaluation of the session

• Authorities and all key stakeholders should

understand and buy into the idea, this can

make the process of NITAG establishment

quite long ( 2-3 years from Kenya experience)

• Participants realized the need to have clear

terms of reference for NITAGs so as to avoid

duplication with other existing committees

especially with the ICC.

• Linkages with existing ad-hoc committees

should be established to create synergies

• The systematic, transparent and independent

process of decision-making used by the NITAG

and the wide range of technical expertise in the

committee justify its place in countries health

systems

• Lack of national experts in certain areas may

limit the NITAG work (South Sudan)

• Unfailing commitment of volunteer experts

throughout their tenure is not guaranteed as

they are busy with other activities

• Learning from other countries will help those

considering setting-up NITAGs speed up

process and learn about how it actually works

in “real life”

9

5.2 Session 2: Guidelines and tools for NITAGs functioning

The objective of the session was to familiarize participants with available tools and guidelines

developed by SIVAC Initiative to assist in an efficient functioning of NITAGs.

Guidelines on NITAGs internal procedures manual development and work plan templates were

distributed to participants. Participants were guided in the reading of the documents before going

into group work. The assignment was to do a review of the draft Kenya and Uganda internal

procedures manuals and workplan, i-e assessing their compliance with the guidelines and

templates presented earlier. This peer- review enabled Kenya and Uganda secretariats to take

note of areas where the documents need improvement; it was also an opportunity for the other

participants to grasp with the guidelines and templates.

Take-away messages from the

session

Participants evaluation of the session

• Role of the secretariat, core members

and no-core members needs to be spelt-

out with great details, and paying

attention to the principle of independence

of the committee

• Independence should be reflected in the

procedures for members nomination

• Clarity in all procedures is key

• Principle of conflict of interest and bias

declaration should be well highlighted in

the manual

• Process for setting-up working group

should follow same guiding principles of

transparency; responsibilities for its

functioning should be clarified

10

5.3 Session 3: Monitoring the committee’s progress and achievements

NITAG draft assessment protocol developed by the SIVAC initiative was presented to

participants. The tool intends to provide guidance for monitoring NITAGs activities as well as

assessing its performance and to a lesser extend its impact. The protocol refers to the 17

standard indicators proposed by WHO and propose an analysis framework for evaluation of

NITAGs with regards to their relevance, effectiveness, management efficiency and sustainability.

Participants provided comments on the assessment tool which will be finalized after additional

field testing. Uganda ACVI shared the lessons learnt from their evaluation exercise conducted in

February 2014

Take-away messages Participants evaluation of the session

• Monitoring and evaluation of

NIATG is made against what

is indicated in the manual

procedures and the workplan

which should reflect MoH

priorities

• Reasons for no

implementation of NITAGs

recommendations should be

investigated even though they

may not lead to the

conclusion of NIATG

inefficiency or failure

• Regular monitoring of

NITAGs activities as well as

scheduled independent

evaluation are means to

identify areas that require

corrective action or capacities

strengthening

05

101520253035404550

Below

Average

Average Above

Average

Comprehensiveness 16 11

Interest 12 15

Clarity 19 8

198

12

15

16

11

Session 3: Monitoring NITAGs progress and

achievements

11

The day ended with the feedback of the Kenya team who attended the US NITAG (ACIP)

meeting in April 2014 (the invitation was facilitated by CDC-Kenya ) followed by the projection of

an edited excerpt of the ACIP proceedings video. The projection illustrated some of the concepts

presented during the sessions: declaration of conflict of interest; framing the question of a

recommendation and organization of working groups.

5.4 Session 4: Development of an evidence-based recommendation - 1) defining the recommendation framework

In this session, participants learnt the 1ststep for developing a recommendation note e-g how to

define the key elements/criteria which will inform the recommendation. The session started with

a presentation of the principles for issuing evidence-based recommendation and a generic

immunization recommendation framework which builds on WHO guidelines for new vaccine

introduction and SAGE criteria. Participants were taken through the concept of framing a

question using the PICO approach (Population/Problem; Intervention; Comparator; Outcome).

Elements which will inform a recommendation should include vaccine-related criteria as well as

health systems, socio-cultural, equity and other issues. The 2nd presentation talked about health

system building-blocks, analysis of the immunization context and how they impact the decision-

making process.

Brainstorming on criteria a

NITAG would consider for

issuing a recommendation

on Rotavirus vaccine :

results from a group work

12

Take-away messages Participants evaluation of the session

• Defining clear recommendation

framework ensure transparency

about the ground for decision

• The outcomes/effects expected

from the vaccine under

consideration should be defined

• Specific information needed to

measure efficacy and

effectiveness should be clarified

• Local data (health systems,

economic, socio-cultural

environment) form an important

part of the recommendation

framework

• Literature review and

consultations with experts and

relevant stakeholders is used to

categorize the importance of

each factor for decision-making

• Different options should be

considered in the

recommendation articulated

according to different elements

that need to be in place, timing,

conditions, and with the aim of

adapting international evidence

to the country context

5.5 Session 5: Development of an evidence-based recommendation - 2) how to search evidence

This session focused on the 2nd step of the development of recommendation note e-g how to

find evidence on the elements of the recommendation framework. The session was organized as

a practice session during which participants had to retrieve evidence on some of the elements

(2-3 elements deemed critical) they have included in the framework of recommendation. The

13

assignment for each group was to register to 2 of the major data bases presented, define

appropriate criteria for literature search, refine search and select relevant articles.

Take-away

messages from the

session

Participants evaluation of the session

• For each critical and

important criteria

listed in the

recommendation

framework, search

should be done 1st for

relevant systematic

reviews and then for

other studies of

acceptable design

0

5

10

15

20

25

30

35

40

45

Below

Average

Average Above

Average

Comprehensiveness 2 7 13

Interest 1 4 17

Clarity 1 8 13

18

13

1

4

17

2

7

13

Session 5: Development of evidence-based

recommendation: 2) evidence search

5.6 Session 6: Development of an evidence-based recommendation – 3) how to evaluate data relevance and quality

The plenary session consisted in the review of the tools proposed for assessing the relevance

and the quality of the data reported in the articles retrieved. The check list and tools presented to

the participants included the GRADE methodology (when doing a systematic review), AMSTAR,

CASP, SIGN and CAQR checklists (for assessing systematic reviews and other types of studies)

The session continued with the groups work: each group was to assess a number of articles on

the topic under study (seasonal flu or rotavirus) using the appropriate check list according to the

study design. Each group presented their report on the data quality assessment exercise.

14

Take-away messages

from the session

Participants evaluation of the session

• The quality of the studies

found needs to be

assessed before they can

be included in the final list

of studies that will be used

to make the

recommendation

• For each criteria a

summary of the quality of

the evidence should be

provided in a table

(Evidence profile)

5.7 Session 7: Development of an evidence-based recommendation – 4) how to structure a recommendation note

The structure of a recommendation note was presented in plenary; this was followed by a

working group session where participants were tasked to review a recommendation prepared by

the Mozambique NITAG in light of the guidelines presented and give their feedback.

15

Take-away messages

from the session

Participants evaluation of the session

• The systematic and

transparent process

used to identify, collect

and assess the

evidence should be

reflected in the write-

up of the

recommendation note

• The recommendation

should include: the

context of the question;

general information on

the question; the

methodology used; the

analysis/judgement of

the evidence; the

proposed

recommendations or

options

0

5

10

15

20

25

30

35

40

45

Below

Average

Average Above

Average

Comprehensiveness 1 10 14

Interest 1 7 17

Clarity 1 12 12

1

12 121

7

17

1

10

14

Session 7: development of evidence-based

recommendation: 4) recommendation structure

5.8 Session 8: Communicating NITAG recommendations

The session included a presentation on principles of effective communication with policy-makers

and an exercise in the form of a role play. For the role play participants were provided with

recommendations on rotavirus and flu vaccination published by WHO and the CDC and asked to

summarize their key points to be communicated to a group supposedly made of a minister of

health, a minister of finance and a key immunisation stakeholder.

Take-away messages from the session

Key principles about communicating with policy-makers: the message should be clear with regards

to:

• The issue addressed by the recommendation ( referring to the PICO question and criteria

for the recommendation

16

• The action required from the audience

• The approaches suggested for undertaking the action

5.9 Session 9: Networking

The format and method of work during the workshop intended to also provide a platform for

initiating collaboration and exchanges between NITAGs. For example the topic on flu vaccine

was selected because 2 participating countries (Uganda and Kenya) are conducting active

surveillance activities on flu vaccine. The working groups were organized to allow the 2 countries

to share perspectives on the issue of flu vaccination and lay down the ground for possible future

collaboration and concerted action on flu recommendation.

In plenary, participants shared their perspectives on a possible mechanism for networking,

information sharing, and peer-to-peer support between NITAGs. Points arising from the

discussion included:

• The network should include all Africa NITAGs; the main goal would be to share

resources and get assistance from each other

• 2-3 days annual meetings could be organized to discuss common issues

• Online forum could be a useful mean to have discussion on technical issues and share

experience

• Role of partners such as SIVAC: assist in the coordination and resources mobilization

6 Conclusions

6.1 Next steps

Each participating country presented the activities planned after the workshop in order to keep

the momentum in starting their NITAGs operations.

17

6.2 Overall evaluation of the training

The graphic below shows the level of satisfaction of participants on some aspects of the training. It is to be noted that the majority expressed high satisfaction in getting new knowledge

++++ General comments from participants:

o Good training, informative and need to be done for others potential members so that a bigger group is trained

o Provide opportunity to get inputs from others countries and review what structures are already in place

++++ Participants suggestions to improve the training:

o Provide the materials to the participants before the workshop and include slides for literature search

++++ Gaps filled through the training:

o Understanding of NITAGs ToR, requirements for establishment of a functional NITAG

o Understanding of NITAGs members role

Very low Very high

18

o Understanding of the need for a strong secretariat

o Knowledge in methods for reviewing studies, PICO approach and formulating recommendation

6.3 Considerations for future trainings

The regional training was also organized with the intention to field test some NITAG capacity

building modules developed by the SIVAC Initiative. The workshop has helped identify revisions

needed in the content of the modules as well as in the roll-out of the training:

++++ Module on evidence quality assessment: will go into greater details on the GRADE methodology which will be taught as a sub-module ( of at least 4 hours)

++++ Analysis of participants evaluations have shown that groups work sessions were the most useful and interesting; future trainings will include more practical exercises; pre-reading materials will be distributed prior to the group work session

++++ Plans for distance support and follow-up in partnership with WHO EVIPNet will be clarified during the workshop

12/04/2014