Technical synthesis report including review, ap-praisal and summary of relevant capacity assess-

ment tools

Technical report prepared by Katharina Stärck,

For FAO

8/02/2013

Objective

The objective of this task was to provide a critical review and synthesis of current, relevant capacity assessment tools.

Approach

The work was based on the rapid assessment already conducted by FAO which was reviewed as part of an earlier deliverable in this project. Rajic (2012) had conducted a rapid assessment of known tools using a set of descriptors to describe individual tools. This framework for de-scribing and assessing tools was modified and applied to the tools considered to be relevant (see below). Additional search efforts were made by using the search terms “capacity assess-ment” and “tool” or “framework” in combination with “trade” in a web browser. This yielded only very few results.

In order to focus the work, the following 2-step check to determine relevance of individual tools was applied:

1) Was the tool developed for appraisals of systems related to sanitary and phytosanitary (SPS) status or to food quality?

a. If no: exclude

b. If yes: proceed to next level

2) Was the tool developed for appraisals of food systems?

a. If no: consider as supportive material, but not included in the detailed compar-ative assessment

b. If yes: include in comparative assessment.

For a tool to be included in the detailed review, the answer to both questions needed to be “yes”. If a tool passed the first selection step, it was considered as supporting material.

The description and assessment of the final list of tools was conducted using a set of charac-teristics. This list was based on earlier work conducted by FAO which was reviewed and re-vised. The characteristics included general objective, target organisation, structure, evaluation process, reporting and implementation. The criteria can be seen in full in Table 2. If an organ-isation had developed several tools or if a tool was available in several versions, these were assessed jointly.

In the discussion, aspects from tools passing only step 1 were also considered.

“Capacity” was defined as “the ability of individuals, organizations and systems to perform functions effectively, efficiently and sustainably” (UNDP, 2008).“ Evaluation” is defined as per FAO charter for the Office of Evaluation as “an assessment...of an activity, project, programme, strategy, policy, topic, theme, sector, opera-tional area, institutional performance, etc. It focuses on expected and achieved accomplish-ments, examining the results chain, processes, contextual factors and causality, in order to understand achievements or the lack thereof. It aims at determining the relevance, impact, effectiveness, efficiency and sustainability of the interventions and contributions of the orga-nizations of the UN system. An evaluation should provide evidence based information that is credible, reliable and useful, enabling the timely incorporation of findings, recommendations and lessons into the decision-making processes of the organization and its members.”1

1 Adapted to become specific to FAO from the “Norms for Evaluation in the UN System”, United Nations Evaluation Group, 2005

2

“Conformity assessment” is a process for demonstrating that features of a product, process, system, person, body or service meet the requirements of standards, regulations and other specifications (UNIDO, 2010). Testing, inspection or certification are typical activities con-ducted as part of conformity assessments. In this context, “evaluation” is mentioned as “the gathering of evidence of conformity” (UNIDO, 2010).

Results and discussion

The list of tools considered in this work is presented in Table 1. Tools from 6 organisations were selected in the first step of the review and tools from 4 organisations included for more detailed review. In the latter, each of the tools was characterised using the characteristics defined above, see details provided in Table 2. In the discussion, also reference is made to tools that were not selected for the second step.

1. Tools included in comparative assessment

All organisations hosting international standardization bodies recognized by the WTO to provide benchmarks in relation to sanitary and phytosanitary issues in international trade are currently providing or planning to provide sector-specific support on capacity assessment to countries. As shown in Table 2, FAO, IPPC, OIE and WHO have developed or are currently developing tools aimed at self-assessment of countries’ current capacity and development needs. These tools are more or less related to the over-arching guidance provided by UNDP (2008). However, different directions in focus and methodological approach have been chosen as described below.

FAO developed a tool for assessing food safety systems (FAO, 2006, 2007) which is line with the UN generic initiative for capacity assessment (UNDP, 2008). This FAO tool is avail-able for either rapid or in-depth assessments. This tool is structured into five modules (Table 3). Laboratories are the only technical capacity that is specifically mentioned. Surveillance is only mentioned in the guide for rapid assessment (FAO, 2007). A process is described for general steps required in order to conduct a capacity assessment. For example, to establish a team of assessors, define the scope of the assessment, gather information, documentation and discussions. Detailed guidance is provided as to how to manage this process. In addition, a set of guiding questions is provided for each module. These provide a general direction of ques-tioning within a module rather than a set of specific points to address or a checklist. For each module, dimensions of capacity are listed. These can be seen as indicators of capacity. But no defined levels of achievement or scoring schemes are provided. The assessment can be con-ducted at either organisational or system level. Methods for the collection of information are also described including literature and documents, stakeholder interviews, self-assessments, workshops or focus groups. Finally, a set of benchmarks for food control management is also provided making reference to relevant international standards.

The tool has been applied in over 30 countries using a participatory approach and through open and engaging dialogue between relevant country-level food safety authorities and FAO. The priorities for capacity development were formulated in collaborative manner, including specific proposed actions for improving the national food control systems (Bessy, personal communication). In addition, several countries have applied the FAO tool independently to assess their food safety systems and to identify development needs (Al-Kandari & Jukes, 2009, 2012; Alomirah et al., 2012; Jia et al., 2013). The application of the approach in these countries appears to have been successful in identifying strengths and weaknesses of the re-spective food safety systems, although the authors do not specifically comment on the useful-ness and practicability of the tool itself. Bagumire et al. (2009) also used the FAO tool as a basis for assessing Uganda’s aquaculture system in terms of international standard compli-

3

ance. They developed an integrated checklist using input also from European Community and US legislation as well as Global GAP2.

In addition to the modular assessment, FAO has also developed a rapid assessment guide in which the capacity of the entire food control system is assessed at once (FAO, 2007). The aim is to allow for a quick and general overview, while the modular approach is meant for coun-tries that already know on which aspect of the food control system they should focus. The two approaches can be used in sequence, stating with the rapid assessment and following-up with an in-depth appraisal of selected components of the food system. This provides flexibility and an option for countries that are at the very beginning of reviewing their food control capacity.

IICA developed a set of tools under the heading “Performance, Vision, Strategy (PVS)” to assess the capacity of national food safety services, national veterinary services and national plant protection organisations (IICA 2007, 2009a, 2009b). All of these tools followed a sim-ilar structure with four fundamental (or basic) components within which 6-8 competencies were listed and assessed qualitatively using 4-6 levels of advancement. According to the focus of the assessment, there were minor differences in the competencies. These tools were seen as part of a process to either raise the awareness of a service for its own strengths and weak-nesses (i.e. “passive mode”) or as a basis for change and development (i.e. “active mode”). The following possible outcomes of an assessment were listed:

Indicate the overall performance in each component, provide relative performance rating each competency, comparison of performance between services in different countries as part of

cooperation or negotiation, identification of differences in performance perceived by stakeholders, foster common understanding to improve, as a basis for cost-benefit decisions for investments by governments or donors, for routine monitoring, identify and specify the needs for grants.

A user manual was made available to facilitate the use of the PVS tools (IICA, 2008). There are no public records of the extent of past or current use of the tools by countries. The use of the IICA PVS seems to have been discontinued (Dr. S. Hutter, personal communication), and the OIE tool (see below) is now used in most Latin America.

IPPC have developed the phytosanitary capacity evaluation tool (PCE). According to leaf-lets available on the internet, this tool was developed to support countries in identifying strengths and weaknesses in relation to current or future phytosanitary systems (IPPC, 2011). The intended user is the national plant protection organisation. The framework and approach is kept flexible such that the country can adapt it to its own priorities. The PCE consists of 13 modules reflecting different components of a phytosanitary system.

The modules include country profile; environmental factors; national phytosanitary legisla-tion; mission and strategy; organisational structure and processes; resources; pest diagnostic capacity; pest surveillance and pest reporting capacity; pest eradication capacity; phytosani-tary import regulatory system; risk assessment; pest-free areas and low prevalence areas; ex-port cetrtification, re-export and transit (R. Clarke, personal communication). For each mod-ule, a set of 1-5 expected outputs as well as qualitative and quantitative indicators are listed.

The user can select which modules should be included in a specific evaluation exercise. Em-phasis is made on stakeholder involvement for which detailed guidance is provided. The pro-posed application also includes a process for consensus building and the development of a national phytosanitary action plan. 2 http://www.globalgap.org/uk_en/

4

This tool has been applied since 2001 and there is therefore extensive experience. A review of the tool has been conducted in 2007 and a revised version was published in 2010. More de-tailed information on the structure of the tool, and the indicators used within each module was unfortunately not available.

The OIE’s tool for the assessment of the performance of veterinary services (PVS) is based on earlier work conducted by IICA (see above) and therefore shows some similarities in the terminology and structure of components. However, the process of application of the tool is very different. It is part of a broader process called the “PVS pathway” with the objective to lead to a sustainable improvement of the veterinary service (Figure 1). Therefore, the assess-ment is only the first step (or “diagnosis” using medical terms) leading to the identification of gaps (“prescription”) and capacity building activities (“treatment”).

Figure 1: Process of the PVS pathway as developed by the OIE (picture from http://www.oie.int/support-to-oie-members/pvs-pathway/)

The PVS pathway operates with third-party assessments managed by the OIE. Funding for visitations is provided by a central fund. The OIE has trained a number of experts in using the tool thus providing the basis for a standardised application of the tool and comparability between country results. A manual is provided to the experts. Missions typically consist of 2-4 experts. At present, 119 countries have requested to be assessed and 82 reports have been finalised3. The tool has been most intensively used in Africa, but also in all other OIE regions. A selection of full reports is available online4. The reports follow a standardised format. For each indicator, the level of advancement is allocated and presented together with the evidence on which this was based, the findings, strengths, weaknesses and recommendations. A total of 46 critical competencies are assessed. Five qualitative levels of advancement are used throughout ranging from level 1 (“basic, no compliance) to 5 (“well advanced”, full compli-3 OIE: PVS evaluation missions. Available online http://www.oie.int/support-to-oie-members/pvs-evaluations/status-of-missions/ [accessed 31.01.2013]4 OIE: PVS evaluation reports. Available online http://www.oie.int/support-to-oie-members/pvs-evaluations/oie-pvs-evaluation-reports/ [accessed 31.01.2013]. The report for Israel (2011) was consulted to describe the report structure and content. Some earlier reports have a different structure.

5

ance). An executive summary and list of recommendations is provided. The assessment is only a first step in the process for capacity building applied by the OIE. The next step consists of a gap analysis. This uses a different tool5 and provides a quantitative evaluation of the country’s priorities for capacity building in order to more fully comply with international standards. It also considers budgetary aspects. To date, 77 countries have required a gap ana-lysis and 33 reports have been finalised6. Some reports are available online7.

WHO is currently developing a self-assessment tool focusing on public health issues in rela-tion to trade. Details on this tool are not yet available, but are expected in June 2013 (Z.U. Mirza, personal communication). It is expected that the tool will be used by country govern-ments to assess their strengths and weaknesses. It will be different from other tools in that it focuses on the specific areas related to public health and trade. The tool is currently under final review. Due to the lack of information an appraisal of this tool could not be conducted.

A comparison across tools showed similarities but also differences (Tables 2 and 3). The OIE PVS tool is more focused on status quo, while other tools are more focused on improvements. This may be due to the fact that in the OIE PVS framework another component – the gap ana-lysis – is used to define the development plan.

All tools have strengths and weaknesses the significance of which will partly depend on the purpose of application and the general objective(s) of an assessment. A tool that is kept open and flexible (e.g. FAO tool) will probably be applicable to a wider range of objectives while a tool that is designed for a specific application (e.g. OIE PVS tool) will be effective for this objective but possibly not for others as it may be too detailed and therefore inefficient. An attempt has been made to describe strengths and weaknesses of the tools reviewed here (Table 4).

Most tools did refer to competencies, and these were grouped under some sort of heading, mostly referred to as “component” (Table 3). Some competencies were covered by all tools (e.g. technical competencies, resources), although some tools were much more detailed and had a number of modules covering what was only one module in other tools, e.g. IPPC. Other headings were present in some, but not all tools, e.g. information, communication and educa-tion. Also interaction with the private sector or stakeholders in general was not consistently mentioned.

Our findings indicate that proposed structures across tools do not vary too much and could probably be combined into a comprehensive yet limited number of components. It appears preferable to have a relatively small number of components (for example 4-5) in order to avoid a fragmentation of competencies. A similar number and type of components were also identified when developing a tool for capacity assessment in public health (Bagley & Lin, 2009). There, four capacity areas were considered: policy environment, resources, pro-grammes, and organisational environment.

The use of qualitative or semi-quantitative scoring is broadly used. However, the number of categories used varies between tools and sometimes even within ranging from 4-6 (Table 2). Also, in some tools the “levels of achievement” are clearly defined while other tools do not define such levels. Because the results of assessments will be likely to be discussed in an in-ternational context, it appears preferable to achieve a degree of comparability between out-

5 OIE: PVS gap analysis tool. Available online http://www.oie.int/support-to-oie-members/pvs-gap-analysis/pvs-gap-analysis-tool/ [accessed 31.01.2013]6 OIE: PVS gap analysis missions. Available online http://www.oie.int/support-to-oie-members/pvs-gap-ana-lysis/status-of-missions/ [accessed 31.01.2013]7 OIE: PVS gap analysis reports. Available online http://www.oie.int/support-to-oie-members/pvs-gap-analysis/pvs-gap-analysis-reports/ [accessed 31.01.2013]

6

comes of food control systems. It therefore seems important to aim for more specific scoring categories. As all existing tools use qualitative scores, this is the preferred option. The OIE PVS tool uses a quantitative assessment in the gap analysis. Whether this is practical, neces-sary or appropriate was not assessed here.

It is noted that assessment exercises are resource-remanding. Expected duration of projects was not provided for all tools, but where they were, it indicated a duration of 2-6 weeks with a significant number of experts and other staff involved resulting in a significant number of person months.

2. Tools and programmes considered as supportive material

There are supplementary tools available which focus on more detailed aspects of food control systems, namely conformity assessment, surveillance, biosecurity, inspection and coordina-tion/communication. Findings for these are provided below.

Many aspects of a food control system need to follow standards, regulations or other specific-ations. They can be related to food safety but also to quality. UNIDO (2010) published a gen-eral document on conformity assessment which is also relevant to aspects of food control. Inspection, auditing and accreditation are all possible approaches to conformity assessment as they compare characteristics of processes, products, systems or persons against an agreed standard. This document describes the elements of the conformity assessment process consist-ing of selection, determination, review and surveillance. Specific types of conformity testing such as inspection or auditing are discussed. They can be part of voluntary or regulatory as-sessment schemes or systems implemented by agencies, private suppliers or the industry. Ref-erences for conformity assessments are set by ISO/CASCO standards and guides. The output of conformity assessment is a declaration or certificate of conformity and can result in a mark of conformity on the product. The latter is often controlled through a license process. UNIDO do provide a set of tools that can be used to support conformity assessment but not specific checklists that would be directly applicable to the appraisal of food control systems. However, specific aspects of conformity assessments are highly relevant such as laboratory accredita-tion. Because conformity assessment is such a wide field, some of the tools discussed above cover certain aspects of conformity assessment. For example, the OIE PVS tool can be seen as assessing conformity with OIE standards.

A tool was developed to specifically assess surveillance systems as part of animal health or food safety programmes (OASIS, Hendrikx et al., 2011). This tool was structured into 10 sec-tions (also referred to as “system attributes”) representing characteristics of surveillance sys-tems: objective, organisational aspects, laboratories, surveillance tools and procedures, data management, training, dissemination and communication. A total of 78 assessment criteria was used across the sections (full list available from Hendrikx et al., 2010). The latter are scored on a scale of 0-3. The results are summarised as graphs for each section and overall radar graph is also provided. The tool was developed to be applied with stakeholder involve-ment. There is supplementary material available such as a questionnaire and scoring guide. The tool was apparently only applied in France and Chad (Hendrikx et al., 2011). Sub-sequently, a revised framework for the evaluation of surveillance systems was developed by Drewe et al. (in press8) providing a process as well as a detailed list of characteristics and in-dicators that can be used to evaluate surveillance systems related to animal health. A specific version of this tool for food safety surveillance is not available although the initial review also included public health surveillance (Drewe et al., 2012). This framework could be adapted to

8 See also http://www.rvc.ac.uk/VEPH/SERVAL.cfm 7

surveillance conducted in food systems. It has so far been applied to three case studies in the UK (Drewe et al., submitted)

A specific tool to assess capacity related to biosecurity is also available (FAO, 2007). This provides a process as well as specific “dimensions” of capacity, including policy framework, legal framework, organisational aspects and communication. It is based on the capacity as-sessment framework developed by UNDP (2008). Emphasis is made on the varying needs of countries in terms of biosecurity development and the interfaces between human, animal and plant health. One aim of the tool is to enhance awareness for biosecurity and to enhance har-monisation and integration. The process envisages an initial assessment, a vision, a gap ana-lysis and a plan for future action to close the gaps. This tool has an explicit link to food safety systems and also links to the risk analysis framework (risk assessment, risk management, risk communication). The main emphasis of the tool is on the process, providing a step-by-step plan. Regarding the assessment of current capacity (step 4), a set of broad questions is provided within the dimensions mentioned above. General topics on which data collection could focus are provided but no detailed list of questions or scoring mechanism. The use of existing sectoral tools is explicitly recommended. Reference is also made to specific methodo-logies suitable for collecting information. The assessment is integrated component of a devel-opment plan and capacity building plan.

USDA/FSIS developed at tool for self-assessment of exporting countries with reference to their equivalence to the US regulatory system. A similar tool is available to establish equi-valence for individual sanitary measures. The approach consists of a list of questions which guides the exporting country in their data collection that is required as one component for the US to establish equivalence. The module for central competent authority consists of two sec-tions, A and B, and largely focuses on inspection post-harvest. Section A covers 6 compon-ents that jointly establish the equivalence of the inspection service. It includes the following: Government oversight, food safety regulations, sanitation, HACCP systems, residue and mi-crobiological testing systems. For each component, a set of criteria are listed with sub-ques-tions and regal references provided. Section B provides a template of a questionnaire and tables for data collection. All questions are linked to components in Section A. The question-naire is structured into the following parts with detailed questions for each: Funding, authority and responsibility, staffing and inspection coverage, training of personnel, administrative sup-port and management, laboratory system, risk analysis and information sharing, use of tech-nical innovation, food defense, ante mortem/slaughter, animal welfare at slaughter, post mortem inspection, consumer protection, supervisory visits, official controls against adultera-tion, sanitation, HACCP, egg products, thermally-processed products, chemical residues, test-ing for faecal contamination, testing for Salmonella, testing for Campylobacter, testing for E. coli O157:H7, ready-to-eat products. While this tool is built to serve a very specific objective, i.e. establishing equivalence with US legislation, some components will be applicable in a wider context. These could be used to inform indicators in a generic tool for the assessment of food control systems.

Another tool, the ASEC tool developed on behalf of DFID provides a framework for institu-tional analysis and strengthening coordination between partners involved in SPS matters (An-onymous, 2012). This tool provides a set of methods that can be used to map the strengths and weaknesses of stakeholders with an interest in coordination. The analysis is designed such that it is possible to only focus on stakeholder involved in the food control system. As an out-put from this activity, stakeholders will be identified, their interests and relationships de-scribed and the process of SPS coordination is documented. The latter has a strong focus on communication, responsibilities and decisions and could therefore contribute an in-depth ana-lysis of these aspects in support of a more general tool. Of specific relevance is the component of this tool that aims at rating the performance of the coordination system. The rating is

8

achieved with help of performance indicators which are grouped into six elements of the sys-tem (policy, legislation & regulatory framework; role of private sector; public sector commu-nication; international participation and representation; cross-sectoral SPS coordination; sec-toral SPS coordination). A six-level scoring scale is used (0-5) of each indicator and an aver-age is calculated for each of the six elements. The results are presented graphically on a radar chart (see Appendix 1).

The Asian-Pacific-Economic-Cooperation (APEC) has developed a conceptual framework and strategy for improving food safety in APEC countries and in global food supply chains (APEC, 2011). This is an instrument used by the APEC Food Safety Cooperation Forum (FSCF)9. This initiative is relevant here because it uses a set of four priority areas for capacity building providing a structure that is similar to some of the tools described above and in Table 3. Keeping in mind that these are priority areas for capacity building and not assessment cat-egories, it is yet interesting to see that some competencies were listed under different headings than in the other tools. For example, some surveillance and risk assessment aspects are listed under information sharing and communication. Recently, this initiative has fed into the Global Food Safety Capacity Building Partnership (GFSP) which held its first conference in 2012. One of the outcomes of this conference was the emphasis of a need to assess countries’ capa-city in food safety.

9 APEC food safety cooperation forum (FSCF). Available online http://www.foodstandards.gov.au/scienceande-ducation/apec/ {accessed 31.01.2013]

9

Table 1: List of capacity assessment tools considered

Tool (grouped by agency in alphabetic order)

Date Focus Selected Step 1

Selected Step 2

ANSES (Hendrikx et al., 2011)OASIS: an assessment tool of epidemiolo-gical surveillance systems in animal health and food safety

2011 Surveillance sys-tems

Yes No

CBDNational capacity self-assessment tool for the convention on biological diversity (CBD)

2010 Environment No -

DFIDAgrifood Standards: Ensuring Compliance Increases Trade for Developing Countries (ASEC)

2012 SPS Yes No

FAOStrengthening national food control sys-tems: guidelines to assess capacity build-ing needs

2006 Food safety Yes Yes

Strengthening national food control sys-tems: quick guide to assess capacity build-ing needs

2007 Food safety Yes Yes

Guide to asses biosecurity capacity 2007 Cross-cutting Yes NoIICAPerformance, vision and strategy (PVS) for food safety

2007 Food safety Yes Yes

Performance, vision and strategy (PVS) for national veterinary services

2009 Animal health Yes Yes

Performance, vision and strategy (PVS) for national plant protection organizations

2009 Plant health Yes Yes

IPPCPhytosanitary capacity evaluation (PCE) tool

2010 Plant health Yes Yes

OIEPerformance of veterinary service (PVS) pathway

2010 Animal health Yes Yes

UNIDOApproach to evaluate conformity assur-ance infrastructure

2010 Conformity as-sessment

Yes No

USDAUSDA/FSIS self-reporting tool (SRT) for foreign meat, poultry and egg products inspection systems

2011 Trade and food safety

Yes No

WB 2 Food safety and agricultural health assess-ments and action plans

Several years

Cross-cutting No2 -

WHODiagnostic tool for analysis and assess-ment of trade and health

Expected 2013

Trade and health Yes -3

1-1More information provided at http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/TRADE/0,,contentMDK:21028381~isCURL:Y~pagePK:210058~piPK:210062~theSitePK:239071,00.html2- This was not considered in the next step as it was found to not really represent a “tool” but a set of reports and case studies on the topic.3- Details on this tool are not yet available, publication is expected by June 2013 (personal communication by Z.U. Mirza, WHO)

10

Table 2: Comparison of capacity assessment tools relevant for food control (in sequential order)

Tool features FAO (2006) IICA (2007) OIE (2010) IPPC (2010)GeneralPrimary objective Assess capacity development

needs in the context of food safety and quality, to be ap-plied at system or organisa-tional level

Establish current level of per-formance and to establish priorities for improvement

Assess performance of na-tional veterinary services and compliance with OIE stand-ards

Evaluate the capacity of plant protection systems to inform plans for phytosanitary capa-city building

Key users National food safety services National food safety services National veterinary services National plant protection or-ganisation (NPPO)

Stakeholder engagement Involvement encouraged and process outlined

Informal recommendation for private-public engagement and stakeholder interactions but not formal process provided.

Informal recommendation for private-public engagement but not formal process provided.

Involvement encouraged and process outlined

Legal basis1 International standard (Codex) None International standard (OIE) NoneStructureCore tool structure and terminology2

“components” with “dimen-sions”

“components” with “compet-encies”

“components” with “compet-encies”

“modules”

Number of core compon-ents2

5 4 4 13 (or selection hereof)

Number of indicators by component

4-7 6-8 6-17 Not known

Number of levels by in-dicator

Not defined 4-6 pre-defined “levels of advancement” for each com-petency

5 pre-defined “levels of ad-vancement” for each compet-ency

Not known

Indicator type Not defined Mostly broad and qualitative Mostly broad and qualitative Not knownEvaluation processAssessment type Structure provided to assess

current capacity, but also how to identify future needs.

Passive mode (awareness and training) or active mode (formal assessment, prioritisa-tion, action plan and invest-ment); qualitative ranking.

Formal assessment with de-scriptive output. For each competency summary in-cludes: findings, strengths, weaknesses, evidence and

Formal assessment with de-scriptive output.

11

Tool features FAO (2006) IICA (2007) OIE (2010) IPPC (2010)recommendations.

Evaluators Self-assessment Self-assessment Third-party (OIE certified)

Self-assessment or consultant team with facilitator and mod-ule coordinators

Guide for evaluators No Yes Yes Not knownUse of additional analysis Gap analysis unknown Gap analysis Resource planning, gap ana-

lysis, national action planDuration Not specified Not specified 2-3 weeks 1.5 monthsReportingSpecified reporting struc-ture

No unknown Yes Not known

Field implementationApproach to stakeholder engagement

Process options provided us-ing a range of formats and methods

Stakeholder workshop and/or focus groups

n.a. Integrated into proposed pro-cess.

Assessment coverage Several publications from Middle East, exact number unknown

Some countries in Central and South America

>100 countries Tool formally evaluated and repeatedly improved. Applied in >90 countries. Available as web tool.

1- Legal basis for the organisation that performs this type of assessment2- See Table 3 for more detail on structure3- n.a.=not applicable

12

Table 3: Overview of structure of capacity assessment tools relevant to food control (in sequential order of publication)

Tool developed by: FAO (2006) IICA (2007) APEC (2009) IPPC (2010) OIE (2010)Terminology used to describe sub-structure within the tool

"modules" "components" "priority areas" “modules” = compon-ents of an effective sys-tem

“components”

Unit headings Food control manage-ment

Mission & strategy, structure and processes

Laboratories Technical Technical resource(Laboratory, food safety, risk analysis, GM food safety, sur-veillance)

Diagnostic capacity, surveillance, risk ana-lysis, eradication capa-city, pest-free and low-prevalence areas

Technical (Laboratory, early detection, quarantine, surveillance, pre-vention & control, food safety, veterinary medicines, residues, emerging risks, risk analysis, innovation, identification & traceability, animal welfare) 14 sections with 17 indicators

Food inspection Inspection and certific-ation

Food legislation Regulatory systems Phytosanitary legislation

Human and finan-cial

Human resource Resources Human, physical, financial (11 sections with 15 indicators)

Information, educa-tion, communication

Interaction private sector

Information and Com-munication

Stakeholder interaction (6 sections with 7 indicators)

Safeguarding pub-lic health and mar-ket access

Import regulation, ex-port certification

Market access (8 sections and indicators)

Country profile [Country information provided in report but not formally part of the assessment]

Environmental factors

13

Table 4: Strengths and weaknesses of tools

Tool (grouped by agency in alphabetic order)

Strengths Weaknesses

FAOStrengthening national food control systems: guidelines to assess capa-city building needs

Integration in UN capacity as-sessment framework

Process description and guid-ance

Flexibility in level of detail and application

Rapid assessment possible as a first step

Emphasis on stakeholder in-volvement

Range of data gathering and analysis methodologies provided

Lack of methodological detail

Lack of scoring system Lack of specific indicators Difficulty to standardise Difficulty to compare

between countries

Strengthening national food control systems: quick guide to assess capacity building needs

IICAPerformance, vision and strategy (PVS) for food safety

Flexible application for a range of objectives

Clear, simple structure Similar tools available also for

other SPS sectors User manual provided

Varying scoring levels between indicators

No specific follow-up pro-cess

Phased out

IPPCPhytosanitary capacity evaluation (PCE) tool

Very comprehensive Flexibility in choice of compon-

ents used Emphasis on stakeholder in-

volvement Process for consensus building Integrated in a development and

action plan Extensively used, evaluated and

revised

Very extensive and time demanding

Difficulty to compare between countries

Details on indicators and scoring system not avail-able

OIEPerformance of veterin-ary service (PVS) path-way

Standardised process Clear structure Specific indicators Uniform scoring system Trained experts, comparability,

quality assured Extensively used and revised Uniform reporting

Lack of flexibility Not designed for self-as-

sessment Resource-demanding pro-

cess requiring gap analysis to inform development plans

Focus on compliance with international standards rather than general capacity building

Stakeholder involvement in the process not envisaged

14

Issues to note and to be discussed

1) The information basis for several tools was limited. The text can be updated and re-vised once more information becomes available (e.g. WHO tool).

2) The structure and indicators (where defined) of tools that were reviewed are not too different.

3) In terms of structure and number of components, a synthesis can be achieved with 4-5 components, and possibly a transversal element for legislation.

4) A list of indicators can be developed for each component using the input from existing systems.

5) The number of indicators between components can vary but should not be too extens-ive for practical reasons.

6) The choice of scoring scale should be evidence-based. A preliminary review of the relevant literature indicates that an even number of scales may be preferable, e.g. 6. This is a similar value as most systems currently use (4-6).

7) Interfaces to systems focusing on more detailed aspects of food control are desirable and should be made explicit. There may be more options for interfacing in areas that were not covered here, e.g. laboratory quality assessment.

8) Interfacing with existing tools such as OIE PVS will be important and should be expli-citly assured for animal-derived food.

9) The use of two versions of the tools (rapid assessment and in-depth) may be useful.

10) Some graphical presentation of results may be desirable.

15

References

Anonymous (2012) Toolkit to strengthen SPS coordination, assess the impact of SPS notifica-tions and analyse the costs and benefits of control measures (ASEC). University of Green-wich, Natural Resource Institute, and DFID.

Al-Kandari, D., & Jukes, D. J. (2009). A situation analysis of the food control systems in Arab Gulf Cooperation Council (GCC) countries. Food Control, 20(12), 1112–1118. doi:10.1016/j.foodcont.2009.02.012

Al-Kandari, D., & Jukes, D. J. (2012). The food control system in Saudi Arabia – Centralizing food control activities. Food Control, 28(1), 33–46. doi:10.1016/j.foodcont.2012.03.030

Alomirah, H. F., Al-Zenki, S. F., Sawaya, W. N., Jabsheh, F., Husain, A. J., Al-Mazeedi, H. M., Al-Kandari, D., et al. (2010). Assessment of the food control system in the State of Kuwait. Food Control, 21(4), 496–504. doi:10.1016/j.foodcont.2009.07.015

Asian-Pacific-Economic-Cooperation APEC (2009) Capacity building priority areas. Avail-able at http://www.foodstandards.gov.au/_srcfiles/Capacity%20Building%20Priorities.pdf [accessed 31.01.2013].

Asian-Pacific-Economic-Cooperation APEC (2011) A conceptual framework and strategy for improving food safety in APEC and in global food supply chains. Available at http://www.-foodstandards.gov.au/scienceandeducation/apec/ [accessed 28.01.2013]

Bagley, P., & Lin, V. (2009). The development and pilot testing of a rapid assessment tool to improve local public health system capacity in Australia. BMC public health, 9, 413. doi:10.1186/1471-2458-9-413

Bagumire, A., Todd, E. C. D., Muyanja, C., & Nasinyama, G. W. (2009). National food safety control systems in Sub-Saharan Africa: Does Uganda’s aquaculture control system meet inter-national requirements. Food Policy, 34(5), 458–467. doi:10.1016/j.foodpol.2009.05.002

Drewe, J.A., Hoinville, L.J., Cook, A.J.C., Floyd, T. & Stärk, K.D.C. (2012) Evaluations of animal and public health surveillance systems: a systematic review. Epidemiology and Infec-tion, 140, 575-590.

FAO (2006) Strengthening national food control systems: Guidelines to assess capacity build-ing needs. Rome, Italy.

FAO (2007) Strengthening national food control systems: A quick guide to assess capacity building needs. Rome, Italy.

FAO (2007) FAO Biosecurity Toolkit. Part 2. Guide to assess biosecurity capacity. Available from http://www.fao.org/docrep/010/a1140e/a1140e00.htm [accessed 28.01.2013]

FAO/WHO (2003) Assuring Food Safety and Quality: Guidelines for Strengthening National Food Control Systems. Paper No. 76, available from http://www.fao.org/docrep/006/y8705e/y8705e00.htm [accessed 07.01.2013]

Hendrikx, P., Gay, E., Chazel, M., Moutou, F., Danan, C., Richomme, C., Boue, F., et al. (2011). OASIS: an assessment tool of epidemiological surveillance systems in animal health and food safety. Epidemiology and infection, 139(10), 1486–96.

IICA (2007) Performance, vision and strategy (PVS) for national food safety services. Avail-able from http://www.iica.int/eng/programs/agriculturalhealth/pages/PublicacionesxLin-ea.aspx?linea=13 [accessed 30.01.2013]

16

IICA (2008a) Performance, vision and strategy (PVS) instrument for national agricultural health and food safety and sanitary and phytosanitary measures services. Available from http://www.iica.int/eng/programs/agriculturalhealth/pages/PublicacionesxLinea.aspx?linea=13 [accessed 30.01.2013]

IICA (2008b) Performance, vision and strategy (PVS) for national veterinary services. 2nd Edition. Available from http://www.iica.int/eng/programs/agriculturalhealth/pages/Publica-cionesxLinea.aspx?linea=13 [accessed 30.01.2013]

IICA (2009) Performance, vision and strategy (PVS) for national plant protection services. 2nd

Edition. Available from http://www.iica.int/eng/programs/agriculturalhealth/pages/Publica-cionesxLinea.aspx?linea=13 [accessed 30.01.2013]

IPPC (2011) Phytosanitary capacity evaluation tool (PCE). Flyer available from https://www.ippc.int/index.php?id=13406 [accessed on 28.01.2013].

Jia, C., & Jukes, D. (2013). The national food safety control system of China – A systematic review. Food Control, 32(1), 236–245. doi:10.1016/j.foodcont.2012.11.042

OIE (2012) OIE recommendations on the Competencies of graduating veterinarians (‘Day 1 graduates’) to assure National Veterinary Services of quality. Paris, France.

Jajic, A. (2012) Preliminary Review of Sanitary-Phytosanitary (SPS) Capacity Assessment Tools”. FAO, Rome (version 1.0 of August 25, 2012).

UNDP (2008) Capacity assessment practice note. Available from http://rbec.undp.org/up-loads/public/File/Capacity_Development_Regional_Training/UNDP_Capacity_Develop-ment_Practice_Note_JULY_FINAL.pdf [accessed on 28.01.2013].

17

APPENDIX

1. Radar chart as used in the ASEC tool (2012)

2.

18