quality assurance of monitoring/surveillance data -...

June 12, 2013

Quality Assurance of Monitoring/Surveillance Data

Takanori UKENA, Ph.D.

[email protected]

Ministry of Agriculture Forestry and Fisheries

Food Safety and Consumer Affairs Bureau

1 MAFF

Contents

Importance of Quality Assurance

1. Laboratory Management, ISO/IEC 17025

2. Use of Validated Analytical method

3. Internal Quality Control and Proficiency

Testing

4. Measurement Uncertainty

5. Evaluation of Analytical Results

Experience of MAFF Japan 2 MAFF

Importance of Quality Assurance

3 MAFF

What is Quality?

“degree to which a set of inherent characteristics fulfils requirement".

⇒ In case of analysis

“delivery of reliable data/results within an agreed span of time under agreed conditions, at agreed costs, and with necessary aftercare". .

The “agreed conditions” should include a specification as to the precision and accuracy of the data

4

(ISO 9000)

MAFF

Quality

Management

Quality

Assurance

Quality Control

(QC)

5 MAFF

Quality Management

(ISO/IEC 17025:2005)

Quality management: coordinated activities to direct and control an organization with regard to quality

6 MAFF

(ISO 9000:2005)

The laboratory shall establish, implement and maintain a management system appropriate to the scope of its activities.

Quality management system: management system to direct and control an organization with regard to quality

Note : generally includes establishment of the quality policy, and quality objective, quality planning, quality control, quality assurance, and quality improvement.

Quality Assurance

Quality assurance:

All those planned and systematic actions necessary to provide adequate confidence that analytical results will satisfy given requirements for quality.

GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009)

7 MAFF

part of quality management focused on providing confidence that quality requirement will be fulfilled

(ISO 9000:2005)

Quality Control

Quality Control:

part of quality management focused on fulfilling quality requirement

8 MAFF

(ISO 9000:2005)

for chemical analysis laboratories

⇒ Internal Quality Control and Proficiency testing.

Why quality assurance is needed?

Prevention of quality

problems

Risk manager need to make decision based on data obtained from other persons, other laboratories, other countries.

Basis of cconfidence in the data

International requirement

9 MAFF

What is reliable analytical results?

Use of validated method

If you don’t use validated methods, only you will know

how good your numbers are.

By a laboratory with Quality Management Internal quality control and proficiency testing

Fit for purpose Data produced by a measurement process enables a user to make technically and administratively correct decisions for a stated purpose.

10

Back ground information/data MAFF

Use of analytical results

•Conformity assessment

Import/export control

•Risk management activities Occurrence data for estimation of exposure

Evaluation of effectiveness of risk management options

•International activities

GEMS/Food, Codex, JECFA, JMPR, etc.

11 MAFF

Inadequate data (example)

The data from Brazil and Uruguay could not be used as they had been obtained by analytical methods with high LODs, ranging from 5 µg/kg to 50 µg/kg, which are inadequate to detect and measure ochratoxin A at 0.94 µg/kg and 0.19 µg/kg, the weighted mean concen-trations found in Europe and the USA in cereals and cereal products, respectively.

5.3 Distribution

(JECFA 47, 2001)

12 MAFF

Codex guidelines for Import and Export

Control testing laboratories

•Compliance with the general criteria in

ISO/IEC 17025

•Participation in appropriate proficiency

testing schemes

•Use of validated method

•Internal quality control procedures

Guidelines for the Assessment of the Competence of Testing Laboratories Involved in the Import and Export Control of Food (CAC/GL 27-1997) 13 MAFF

Food Control Laboratory Management:

Recommendations

•International Harmonized Protocol for the Proficiency

Testing

Pure & Appl. Chem., 65 (1993) 2132-2144

•Protocol for the Design, Conduct and Interpretation of

Method Performance Studies

Pure & Appl. Chem., 67 (1995) 331-343

•Harmonized Guidelines for Internal Quality Control

Pure & Appl. Chem., 67 (1995) 649-666

Food Control Laboratory Management: Recommendations (CAC/GL 28-1995, Rev.1-1997)

Protocols and Guidelines adopted in Codex

14 MAFF

Guidelines for Settling Disputes over

Analytical (Test) Results

the following information should be shared between competent authorities of the importing and exporting country

(CAC/GL 70-2009)

• validation status of the analytical method used

• raw data

• internal quality assurance/control

(control charts, sequence of analysis, blank data, recovery data, etc.)

•performance in relevant proficiency testing or collaborative studies;

• official accreditation status of the laboratories.

15 MAFF

Topic １

Laboratory Management,

ISO/IEC 17025

16 MAFF

Why is ISO/IEC 17025 important?

Requirements of Codex guidelines

ISO an International standard / international

recognition

Benefit of accreditation for laboratory

•A marketing advantage for laboratory

often required for international trade

•A bench mark for performance

In depth assessment

Required periodic reassessment

An indicator of technical competence 17 MAFF

Requirements of ISO/IEC 17025

Management (15)

•Organization

•Management system

•Document control

•Review of requests/contracts

•Subcontracting of tests

•Purchasing services/suppliers

•Service to the customer

•Complaints

•Control of non-conforming tests

•Improvement

•Corrective action

•Preventive action

•Control of records

•Internal audits

•Management review

Technical (10)

•General

•Personnel

•Accommodation and environmental conditions

•Test method and method validation

•Equipment

•Measurement traceability

•Sampling

•Handling of test and calibration items

•Assuring quality of test and calibration results

•Reporting the results

18 MAFF

What is Accreditation of ISO/IEC 17025?

Formal recognition by an authoritative body of the competence of a

laboratory to carry out specific tests or types of tests in a reliable,

credible and accurate manner

Evidence of Quality, Credibility and Reliability

•technical competence of staff

•validity and appropriateness of test methods

•suitability, calibration and maintenance of equipment

•appropriate handling of test materials

•quality control procedure, e.g., use of CRM, Proficiency testing, etc.

Actual performance and ethical behavior is critical. Not a guarantee

19 MAFF

Accredited

to each analytical method

each laboratory Need to confirm :

For what method the laboratory is accredited?

Which laboratory is accredited?

If the organization has blanch laboratories.

Notes for Accreditation of ISO/IEC 17025

20 MAFF

ILAC, APLAC

ILAC: International Laboratory Accreditation Cooperation

The international organization on laboratory accreditation with membership of accreditation bodies and affiliated organizations.

Maintain a Mutual Recognition Agreement (MRA) among accreditation bodies to facilitate trade and to eliminate the need for duplicate accreditation.

Assist developing accreditation programs

APLAC: Asia Pacific Laboratory Accreditation Cooperation

21 MAFF

ILAC-MRA, APLAC-MRA

A laboratory accredited by one MRA partner has

equivalent competence to laboratory accredited by the

other MRA partner(s)

https://ilac.org/home.html http://www.aplac.org/home.html

Accreditation body shall comply with ISO/IEC 17011 and that its accredited facilities are in compliance with ISO/IEC 17025 (laboratories), ISO 15189 (medical laboratories) and ISO/IEC 17020 (inspection bodies).

22 MAFF

“Tested once, accepted everywhere”

https://ilac.org/home.html

https://ilac.org/home.html

http://www.aplac.org/home.html

http://www.aplac.org/home.html

Topic 2

Use of Validated Analytical method

23 MAFF

Chemical Analysis

Qualitative

•the presence or absence of a particular substance, but

not the mass or concentration.

•Identification of the substance

e.g., structure elucidation of natural product

Quantitative

•determination of the absolute or relative abundance

(concentration) of particular substance(s) present in a

sample. 24 MAFF

Analytical method for Risk

Management

Analytical Methods

•Whenever

•Whoever (analytical specialist)

can follow the method procedure and obtain

similar results for the same sample.

(e.g., variability of results fall in expected interval).

25 MAFF

Validated method

“If you don’t use validated methods, only you will

know how good your numbers are.” AOACI

Validated Test Method: An accepted test method for which validation studies have been completed to determine the accuracy and reliability of this method for a specific purpose. Reference: ICCVAM Guidelines for the nomination and submission of new, revised and alternative

Guidelines on Analytical Terminology (CAC/GL 72-2009)

26 MAFF

Validated method is a method tested in many laboratories on the same samples, showing that the method is rugged enough to produce comparable results in different labs, with different operators.

Method Validation Study

Collaboration study, Ring-test- trial

27 MAFF

Validation Study Protocol and Guidelines

AOAC International AOAC Official Methods of Analysis (2002). Interlaboratory Collaborative Study, Appendix D: Guidelines for Collaborative Study Procedures To Validate Characteristics of a Method of Analysis http://www.aoac.org/vmeth/guidelines.htm

ISO 5725-2:1994 Accuracy (trueness and precision) of measurement methods and

results -- Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method

IUPAC Protocol for the Design, Conduct and Interpretation of Method-Performance Studies. Pure & Appl. Chem., 67(2), 331-343 (1995)

28

http://www.iupac.org/publications/pac/67/2/0331/

MAFF

Comparison of Protocol for the study

* test statistics and application procedure are different between the protocols . 29 MAFF

General Criteria in Codex Procedure Manual

Single-laboratory Validated Method

single-laboratory validated methods must fulfill the following criteria:

(i)the method is validated according to an internationally recognized protocol (e.g. IUPAC Guidelines)

(ii) the use of the method is embedded in a quality system in compliance with the ISO/IEC 17025

30 MAFF

The method should be complemented with information on accuracy demonstrated for instance with:

–regular participation in proficiency schemes, where available;

–calibration using certified reference materials, where applicable;

–recovery studies performed at the expected concentration of the analytes;

–verification of result with other validated method where available.

31 (Codex Procedural Manual)

MAFF

Harmonized Guidelines for Single-Laboratory Validation of Methods of Analysis

IUPAC/ISO/AOAC, 2002

Guidelines on Single-Laboratory Validation

Performance characteristics

•Applicability •Selectivity •Calibration and linearity •Trueness, Precision •Recovery •Limit of Detection, Limit of Quantification •Sensitivity •Ruggedness

etc.

32 MAFF

Recommended methods in Codex

Recommended Methods of Analysis and Sampling (CODEX STAN 234-1999)

General Methods of Analysis for Contaminants (CODEX STAN 228-2001)

33 MAFF

Criteria approach

Identify method performance characteristics based on existing method validation data.

and establish criteria for evaluating acceptable method of analysis

⇒ laboratory can choose a method meeting criteria (flexibility)

34 MAFF

Guidelines for Criteria approach

<Codex Procedural Manual>

Working Instructions for the Implememtation of

the Criteria Approach in Codex

Guidelines for Establishing Numeric Values for

Method Criteria and/or Assessing Methods for

Compliance Thereof

35 MAFF

Examples for numeric values for the criteria

* The sR should be calculated from the Horwitz / Thompson equation 36 MAFF

Terms related to variability of

analytical results

37 MAFF

Terms related to variability of test results

Accuracy

Trueness Precision

Repeatability Reproducibility

Measurement Uncertainty

Intermediate precision

38 MAFF

Distribution of results for repeated analysis

m

A

C µ: True value m: mean X-axis:

concentration Y-axis: frequency

B D

µ µ

µ µ

m

m m 39 MAFF

accuracy, trueness, precision

accuracy

trueness

precision

○

○

○

×

○

×

×

×

○

µ (true value); m (mean); σ (standard deviation); c (m+s)

m m

σ

m

s

40 MAFF

A B C

Accuracy

The closeness of agreement between a test result or measurement result and a reference value.

•When applied to a test method, the term accuracy refers to a combination of trueness and precision.


41 MAFF

A reference value is usually provided with reference to a) a certified reference material; b) a reference measurement procedure; c) a comparison of measurement standards.

Trueness The closeness of agreement between the average

of an infinite number of replicate measured

quantity values and a reference quantity value.

•Measurement trueness is inversely related to systematic measurement error, but is not related to random measurement error.

• Measurement accuracy should not be used for ‘measurement trueness’ and vice versa.


42 MAFF

Precision The closeness of agreement between independent

test/measurement results obtained under

stipulated conditions.

•depends only on the distribution of random errors and does not relate to the true value or to the specified value.

•usually expressed in terms of imprecision and computed as a standard deviation of the test results. Less precision is reflected by a larger standard deviation.

•Quantitative measures of precision depend critically on the stipulated conditions. Repeatability and reproducibility conditions are particular sets of extreme conditions.

Guidelines on Analytical Terminology (CAC/GL 72-2009) 43 MAFF

Repeatability

Precision under repeatability conditions.

Repeatability conditions:

Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in the same test or measuring facility by the same operator using the same equipment within short intervals of time.


Reproducibility

Precision under reproducibility conditions.

Reproducibility conditions:

Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in different test or measurement facilities with different operators using different equipment.


Relative standard deviation

RSDｒ : Repeatability relative standard deviation

RSDR : Reproducibility relative standard deviation

Relative standard deviation (%)

= standard deviation (SD) / mean ×100

same as CV (coefficient of variation)

1

221

++

n

)μx()μx(SD n

46 MAFF

Horwitz equation

RSDR （%）＝ 2 C -0.1505

RSDR （%）: reproducibility relative standard deviation

C：concentration ratio

if concentration ratio =100 %, then C = 1

1 %, C = 0.01

A ppm C = A×10-6

C.-

R RSD 10log5012(%)

Horwitz equation

approximation of Horwitz equation

47 MAFF

Graphical expression of Horwitz equation

48 MAFF

-200

-150

-100

-50

0

50

100

150

200

相対標準偏差[%]

1 ppm 1 ppb 1 ppt 1%

16% 45%

128%

4%

RS

DR

Horwitz/Thompson equation

RSDR （％）＝

Thompson, M., Analyst, 125, 385-386 (2000)

22

C-0.5

2 C-0.1505

C < 1.2×10-7

C > 0.138

1.2×10-7 ≤ C ≤ 0.138

RSDR （%）: Reproducibility relative standard deviation

C：concentration ratios

49 MAFF

Thompson, M., Analyst, 125, 385-386 (2000)

-30

-20

-10

0

10

20

30

Concentration

Relative

standard de

viation[%

]

1 ppt 1 ppb 1%

16%

22%

1 ppm

0.1ppm

Graphical expression of

Horwitz/Thompson equation

50 MAFF

HorRat Value

HorRat(R) = RSDR/PRSDR HorRat(r) =RSDr/PRSDR

RSDr, RSDR: observed relative standard deviation

(sr/C, sR/C)

PRSDR: predicted relative standard deviation

calculated from Horwitz/Thompson

Normal range of HorRat

The ratio of the reproducibility relative standard deviation to that calculated from the Horwitz equation,


Limit of Detection (LOD)

and

Limit of Quantification (LOQ)

52 MAFF

What is Limit of Detection?

Limit of Detection (LOD) :

The true net concentration or amount of the analyte in the material to be analyzed which will lead, with probability (1-β), to the conclusion that the concentration or amount of the analyte in the analyzed material is larger than that in the blank material.

Critical Value (Lc) :

The value of the net concentration or amount the exceeding of which leads, for a given error probability α, to the decision that the concentration or amount of the analyte in the analyzed material is

larger than that in the blank material.

53 MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009)

LOD and Critical Value (Lc)

Lc Results of blank

α: Type I error (false positive)

β: Type II error (false negative)

0

Results of LOD concentration sample

LOD

Sample of which concentration > LOD will lead to the decision that the concentration of the material is larger than that in the blank material with probability larger than 1-β

If results > Lc, then they will lead to the decision that it is not blank, detected with probability larger than 1-α.

If results are lower than Lc, they are N.D. (could not decided to be detected)

54

β

MAFF

The limit of detection LOD is estimated by,

LOD ≈ 2×t1-αν σo [where α = β], —Where

t1-αν : Student's-t, based on ν degrees of freedom for a one- sided confidence interval of 1-α

σo : the standard deviation of the true value (expectation).

LOD = 3.29 σo, when the uncertainty in the mean (expected) value of the blank is negligible, α = β = 0.05

Usually estimated by using variance of sample (SD of True value is not known.) LOD depends on probability (α and β) Calculate t-value depending on number of data

(LOD should not always equals to 3.29×s (SD. observed))

55 MAFF

α=0.05:

Degree of Freedom = 1 ⇒ t 0.95,1 = 6.31 (2×t-value=12.62)

Degree of Freedom = 10 ⇒ t 0.95,9 = 1.81 (2×t-value=3.62)

Degree of Freedom = ∞ ⇒ t 0.95,∞= 1.64 (2×t-value=3.29)

t-distribution, df=１

t-distribution, df=3

t-distribution, df =10

Standard normal distribution

t-Distribution and Standard Normal Distribution

56 MAFF

Limit of Quantification

Limit of Detection (LOD) : A method performance characteristic generally expressed in terms of the signal or measurement (true) value that will produce estimates having a specified relative standard deviation (RSD), commonly 10% (or 6%).

LOQ is estimated by:

LOQ = kQ ×σQ , —Where kQ = 1/RSDQ

If σ is known and constant (SD of the estimated quantity is independent of concentration) , and σQ = σo :

substitution of kQ = 10% ⇒ LOQ = (10×σQ) = 10 σo

（3.04 ×LOD, α = β = 0.05）

57 MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009)

Example of Method Criteria at Different Concentrations

58 (Codex Procedural Manual) MAFF

Topic 3

Internal Quality Control and

Proficiency testing

59 MAFF

3-1

Internal Quality Control (IQC)

60 MAFF

Harmonized Guidelines for Internal Quality Control in Analytical Chemistry Laboratories, Pure & Appl. Chem. 67 (1995) 649-666.

Introduction Definition Quality assurance practices and internal quality control Internal quality control procedure IQC and within-run precision Control materials in IQC Recommendations Conclusions

Guidelines on internal Quality Control

61 MAFF

Definition

Internal Quality Control

Internal Quality Control：

Set of procedures undertaken by laboratory staff for the continuous monitoring of operation and the results of measurements in order to decide whether results are reliable enough to be released.

62 MAFF

Control materials in IQC

Control materials are characterized substances that are inserted into the run alongside the test materials and subjected to exactly the same treatment.

•appropriate concentration of the analyte

•same matrix in terms of bulk composition, including minor constituents

•similar physical form

•stable and possible to divide the material into effectively identical portions for analysis

63 MAFF

Certified reference material (CRM):

Reference material accompanied by documentation issued by an authoritative body and providing one or more specified property values with associated uncertainties and traceability, using valid procedures

Documentation is given in the form of a “certificate” (ISO guide 30:1992). Procedures for the production and certification of certified reference materials are given, e.g. in ISO Guide 34 and ISO Guide 35.

Use of Certified reference material (1)

GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009) 64 MAFF

Metrological Traceability:

Property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the stated measurement uncertainty.

Metrological traceability of a measurement result does not ensure that the measurement uncertainty is adequate for a given purpose or that there is an absence of mistakes.

Metrological Traceability


Comparability of measurement result, for quantities of given kind. (metrologically traceable to the same reference)

Metrological Traceability (2)

Definition of unit, SI base

Primary national standard

Reference standards

Working standards

Measurement result

BIPM

Primary national standard (other countries)

End user

National Metrology Institute, Designated Institute

Calibration organization

Competent authority, Industry, Academia

Larger Measurement Uncertainty in lower stage 66 MAFF

Use of Certified reference material (2)

COMAR (International Database)

http://www.comar.bam.de/en/ Purchase from reagents manufacturer or reagents sales company

Comparison of the measurement results with the certified value

67 MAFF

http://www.comar.bam.de/en/

http://www.comar.bam.de/en/

Comparison of the measurement results with the certified value

CRMmm CC Δ

http://www.erm-crm.org/ERM_products/application_notes/Pages/index.aspx

22

Δ CRMm uuu +

Δm：absolute difference between mean

measured valued and certified value Cm: mean measured value CCRM : Certified Value

UΔ： expanded measurement uncertainty

um: measurement uncertainty uCRM : uncertainty of the certified value

If Δm ≤ UΔ then there is no significant difference between the measurement result and certified value.

ΔΔ 2uU

68 MAFF





Constraint on the use of CRM

⇒ Preparation of House reference material e.g. materials in proficiency testing, spiked control materials, etc.

For the majority of analysis there is no closely matching CRM available.

Cost to stock every relevant CRMs

Not applicable to unstable materials

Availability of sufficient amounts for IQC to use over extended periods

69 MAFF

If the use of reference material is not practical, limited check on bias is possible by a test of recovery.

Recovery checks

A test portion of the test sample spiked with a known amount of the analyte is analyzed alongside the original test material.

applicable unstable analyte/matrices, ad hoc analysis

70 MAFF

Blank determination

an analysis of a sample without the analyte, or an analysis without a sample, i.e. going through all steps of the procedure with

the reagents only.

1.In many analyses sample results are calculated by subtracting blank readings from sample readings.

2.Blank readings can be excellent monitors in quality control of reagents, analytical processes, and proficiency.

3. They can be used to estimate several types of method detection limits.

71 MAFF

Insertion of control materials for IQC (1)

Recommendation

1.Short (e.g. n<20) frequent runs of similar materials

2. Longer (e.g. n>20) frequent runs of similar materials

•at least once per run •analyze in duplicate at least half of the test materials selected at random •insert at least one blank determination

•at an approximate frequency of one per ten test materials at least once per run •analyze in duplicate at minimum of five test materials selected at random •insert one blank determination per ten test materials

72 MAFF

3. Frequent runs containing similar materials but with a wide range of analyte concentrations

4. ad hoc analysis (statistical control is not applicable)

Insertion of control materials for IQC (2)

•at an approximate frequency of one per ten test materials at least once per run.

•at least two concentration levels, one close to the median level of typical test materials, and the other is approximately at the upper or lower of decile as appropriate.

•duplicate a minimum of five test materials.

•insert one procedural blank per ten test materials.

•duplicate analysis on all of the test materials •recovery tests or use of formulated control material •blank determination

73 MAFF

Use of Control Charts

74

Control Chart (Shewhart Chart):

A statistical tool used to monitor process stability and control. One of the Seven Basic Tools of Quality Control.

MAFF

1. Understanding current and past process performance and its degree of consistency

2. Establishing a "state of statistical control" by identifying and removing causes of unnatural (or "special cause") variation so as to achieve a consistent and predictable level of process quality over time;

Common Control Charts

75

chart R- X

chart s-X

chart R-X

MAFF

How to calculate/plot X-Bar and R chart (1)

76 MAFF

77

: The grand mean of all subgroup averages X

X : The mean for each subgroup

n

XXXX n+++

21

n : The number of measurements within a subgroup (here, n=2)

k

XXXX k+++

21

k : The number of subgroups (here, k=9)

: Calculate the following statistics


MAFF

78

R : The average of the ranges for all subgroups

k

RRRR k+++

21

Ri : The individual range for each subgroup

k : The number of subgroups (here, k=9)


MAFF

79

Upper control limit (UCL):

RAXUCL 2+

: Calculation of Upper and Lower control limit

Lower control limit (LCL):

RDUCL 4

For

For R chart:

X chart:

RAXLCL 2

RDLCL 3

For

For R chart:

X chart:


MAFF

80

Constants for Calculating Limits for X-Bar and R charts


http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc321.htm MAFF

81 MAFF


Time line (Day/Run)

82

Example of X-Bar and s chart

MAFF

Time line (Day/Run)

3-2

Proficiency testing

83 MAFF

Proficiency testing (PT) :

evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons

External Program for Quality Control (Proficiency testing)

84

(ISO/IEC 17043:2010)

•Participating laboratories analyze same sample distributed from proficiency testing provider.

•Reported results were compared to the assigned value and statistically evaluated.

•Participants can confirm their results.

MAFF

How to participate in PT program?

Proficiency testing provider

Internationally renowned: Fera(FAPAS, FEPAS),

AOACI, AOCS, etc.

Domestic (In case of Japan): JSAC, FDSC, JAB, etc.

Participation

Through internet, agency, etc.

Check schedule of individual proficiency testing round.

85 MAFF

example ：FAPAS

• Round: Nutritional components, Food ingredients

•contaminants, Pesticides, Veterinary drug residues,

Food additives, etc.

•Participating laboratories analyze distributed sample

using their method. Report the results and the method

information through website.

•Results were Statistically evaluated and z-score were

returned to participating laboratories.

|z|≦2, 2＜|z|＜3, |z|≧3

Example of Proficiency Testing Scheme

86 MAFF

Result of PT

ｚ = (x - X) / σp x: the participant’s reported result

X：the assigned value e.g., robust mean after excluding results that are clearly spurious and outliers by statistical evaluation.

σp ：the target value for standard deviation

e.g., using RSDR from collaborative study, calculation from Horwitz equotion etc.

Usually shown in z-score defined as follows:

87 MAFF

z-Scores for Patulin (80.7 g/L) in Apple Purée Test Material

From: FAPAS PROTOCOL FOR THE ORGANISATION AND ANALYSIS OF DATA SIXTH EDITION, 2002

Graphical expression of z-Scores (example)

88 MAFF

How to interpret a result of PT?

Checking bias of the results

Review of method procedure and management system of

laboratory by periodical participation

Information on methods used by other laboratories

Importance of Participating PT

•A result of |z| >2 is not specially rare. •Is that a usual operation for the laboratory?

z-Scores “|z| ≤ 2” is “satisfactory” ?

89 MAFF

Topic 4

Measurement Uncertainty

90 MAFF

Definition

Non-negative parameter characterizing the dispersion of the values being attributed to a measurand, based on the information used.

Measurement Uncertainty in Codex (1)

Expanded measurement uncertainty:

Measurement uncertainty:

product of a combined standard measurement uncertainty and a factor larger than the number one


Estimated range in which the true value would be found

Showing reliability of the results

Estimated by experimental data and statistical evaluation

What is Measurement Uncertainty?

92 MAFF

Analytical results and its variability

•Analytical results of repeated analyses of the

same sample can be found in a normal

distribution.

•In case of chemical quantitative analysis,

reproducibility, between laboratory variation,

depends on concentration of analyte,

independent of variety of food or analyte.

93 MAFF

68%

95%

99%

68% of the observations have values within the range of µ±1σ

95% of the observations have values within the range of µ±2σ

Normal Distribution

Mean : µ

SD : σ

+σ +2σ +3σ +4σ − σ − 2σ − 3σ − 4σ µ

94 MAFF

Measurement Uncertainty and Bias

(Value we want to know)

(unknown)

Population distribution (unknown)

True Value

Sample

distribution

Mean

Bias: difference between the result and the true value

Measurement Uncertainty : Estimated range in which the

true value would be found.

X-axis: Concentration

Y-axis: Density

95 MAFF


Guidelines on Measurement Uncertainty (CAC/GL 54-2004)

Measurement Uncertainty have to be estimated in Codex.

One of the requirements of the ISO/IEC 17025:2005

The measurement uncertainty of a result shall be estimated and then made available if requested.

Guidelines CAC/GL 27-1997 require laboratories involved in the import/export of foods to comply with general criteria in ISO/IEC 17025.

96 MAFF


Reporting the results in a form of “a ± U”


—where a: The best estimate of the true value of the concentration

of the measurand (the analytical result) u: The Standard uncertainty U: The expanded uncertainty (usually equal to 2u).

The range “a ± 2u” represents a 95% level of confidence

in which the true value would be found.

97 MAFF


2. Top-down approach

Use data from collaborative trials, proficiency studies, validation studies or intra-laboratory quality control samples,

1.Bottom-up approach

Identification of uncertainty sources

Estimation of the size of the uncertainty component associated with each potential source.

Calculation of combined uncertainty

Estimating Measurement Uncertainty

98 MAFF

Typical Value of Expanded Uncertainty

CAC/GL 54-2004 GUIDELINES ON MEASUREMENT UNCERTAINTY 99 MAFF

MU in Compliance Decisions

Maximum Level

Result ±U

above ML

Result > ML but ML within U

Result < ML but ML within U

Result ±U

< ML

＋U

－U

Guidelines on Measurement Uncertainty (CAC/GL 54-2004) Guidelines on Estimation of Uncertainty of Results (CAC/GL 59-2006)

Situation ii） iii） iv） i）

Result

100

MAFF

Topic 5

Evaluation of Analytical

Results

101

MAFF

Information from Laboratory

Require all relevant data (especially when using 3rd Party laboratory)

•validation status of the methods of analysis used, sample

handling, preparation procedures.

•raw data (including spectral data, calculations, chemical

standards used);

•results of repeat analysis;

•internal quality control (control charts, blank data, recovery

data, uncertainty data, use of RMs);

•performance in proficiency testing

•official accreditation status of the laboratory

102

MAFF

Typical Problem for the Reported

Results (Examples)

•mistakes

e.g., sampling number, classification of commodity

•Ambiguous data

no formula for calculations of LOD, LOQ, recovery, etc.

no information on recovery correction

•Inappropriate significant figures

103

MAFF

Check

Analytical work was correctly done as specified?

•Sample treatment was appropriate ?

•Validated method used? Verified for the commodity?

•LOD and LOQ were properly calculated?

•LOQ was lower than expected occurrence level?

•Calibration was appropriate?

–reliable standard reagents

–linearity of calibration：R2≧0.99 ?

•Recovery corrected?

•Significant figures were properly calculated?

104

MAFF

Recovery

•Recovery/recovery factors:

Proportion of the amount of analyte, present in, added to or present in and added to the analytical portion of the test material, which is presented for measurement.


105 MAFF

Cf : fortified concentration, analytical results of spiked sample

Cu : unfortified concentration, analytical results of un-spiked sample

CA : added concentration

Food /Feed

Reagents spiked

Measurement Results

106

A

uf

C

CC(%) Recovery Marginal

MAFF

UA

f

CC

C Recovery % Total

+

107

Criteria for Recovery in Codex

(Codex Procedural Manual) 107 MAFF

Recovery Correction

All data, when reported, should

(a)be clearly identified as to whether or not a recovery correction has been applied and (b) if a recovery correction has been applied, the amount of the correction and the method by which it was derived should be included with the report.

This will promote direct comparability of data sets. Correction functions should be established on the basis of appropriate statistical considerations, documented, archived and available to the client.

Harmonized IUPAC Guidelines for the Use of Recovery Information in Analytical Measurement (CAC/GL 37-2001)

108 MAFF

Recovery check as IQC

IQC control charts for recovery should be established during method validation and used in all routine analysis. Runs giving recovery values outside the control range should be considered for re-analysis in the context of acceptable variation, or the results reported as semi-quantitative.

Harmonized IUPAC Guidelines for the Use of Recovery Information in Analytical Measurement (CAC/GL 37-2001)

109 MAFF

Significant Figures(1)

•Addition and Subtraction

–The result should have as many decimal places as the

measured number with the smallest number of decimal

places.

(e.g.) 2.54 cm + 15.75 cm = 18.29 cm

1.46 mg + 12.7 mg = 14.2 mg

(e.g.) 3.58 kg – 0.779 kg = 2.80 kg

1.46

+ 12.7

14.16 2

110 MAFF

Significant Figures(2)

•Multiplication and Division

–The result should have as many significant figures as the measured number with the smallest number of significant figures.

e.g. 1.42 mg/kg × 15 kg = 21 mg 100 m ÷ 12 s = 8.3 m/s

Divided by 4 pieces

8.43cm ÷ 4 = 2.11 cm

1.42

× 15

710

142

2130

111 MAFF

Exact numbers, such as the number of people in a room, have an infinite number of significant figures.

Rounding of numbers (1)

ISO 80000-1: 2009

Rounding

range： 0.1

Rounding

range：10

112 MAFF

The rounding shall always be carried out in only one step. e.g. 12.254 should be rounded to 12,3 and not first to

12,25 and then to 12,2.

Rule A: The even multiple

Rule B: The greater in magnitude

ISO 80000-1: 2009 (ISO 31-0 1992)、JIS Z 8401:1999

If there are two successive integral multiples equally near the given number, two different rules are in use.

113 MAFF

Rounding of numbers (2)

Experience of MAFF Japan

114 MAFF

Roles of MAFF and MHLW for food safety

Processing, Distribution Primary Production

Processed Food

Fresh produce Raw material

Food Chain

MHLW (enforcement)

Setting MRLs

Food Inspection, etc.

MHLW (enforcement) Import quarantine Exporting Country

MAFF (enforcement) Fertilizer, Feed, Veterinary drug the use of Pesticide

Cooperation

MAFF (recommendation)

Improvement of production/processing methods of domestically produced food, etc.

Imported food

Consumer Farm

Domestically produced Food

115 MAFF

【Contents】

・Identification of a food safety problem

・Establishment of a risk profile

・Ranking of the hazard for risk management priority

・Establishment of risk assessment policy

・Consideration of the result of the risk assessment

・ Evaluation of risk management options

・ Implementation of management decision

・Risk communication etc

Standard Operating Procedure (SOP)

for MAFF & MHLW

Publication：25 Aug 2005, rev 2006, 2012

116 MAFF

Example of priority list of Chemical

hazards

for food safety (MAFF 2006, revised in

2010) •Metals

–Arsenic

–Cadmium

–Lead

–Methyl mercury

•Mycotoxins

–Aflatoxins

–Zearalenone

–T2-toxin

HT-2 toxin

•Marine toxins –Paralytic shellfish poisoning toxins

–Diarrhetic shellfish poisoning toxins

–Ciguatera poisoning toxins

–Domoic acid

–Brevetoxin

•Process contaminants –Acrylamide

–Polycyclic aromatic hydrocarbons

–Furan

–3-monochloropropane-1,2-diol

(3-MCPD)

3-MCPD fatty acid esters

etc. 117 MAFF

Guideline for the evaluation and publication of analytical results of surveillance/ monitoring (2006, MAFF)

Purpose To Guide MAFF officials for assessing the quality of analytical reports from internal/private laboratories, and for relevant considerations for their publication To provide analytical laboratories with the format for reporting the result of analysis

118 MAFF

Guideline for assessing the result of surveillance/ monitoring (MAFF)

Format for reporting the result of analysis by Laboratories

Laboratory Lab Name, Address, Person in charge

A . Analysis

1. Analyte / Matrix

2 . Laboratory Sample Date of receipt, Condition for storage of laboratory sample and

replicate sample (temperature etc.)

3.Sample Preparation Procedure for preparation of a portion for the measurement from Laboratory sample

119 MAFF

4. Information of analytical method Standard operating procedure of the analytical method

5. Information of method validation Multi Lab / Single Lab / Other

Matricies used for validation

6. LOD, LOQ Value, Calculation method

7. Calibration Used standard reagent, Purity, Protocol for preparation of

standard solution, Linear range

120 MAFF

8. Recovery Matrix, Analyte, Concentration, Number of replication, Value of

recovery rate and RSDr, Raw data or corrected by recovery

9. Measurement Uncertainty

Value of measurement uncertainty Method for estimation

10. Analytical results Number of tested samples, Value of measurement of each sample

121 MAFF

B Quality Control 1. Internal in accordance with GLP guideline by MHLW frequency 2. Proficiency testing provider, matrix, analyte, z-score

※ must participate in the last two years

(same or similar matrix and analyte) 3. ISO/IEC17025 Accreditation

122 MAFF

C Sampling (only when the laboratory itself conducted sampling) Sampling method and it’s reference Sample size for each step in the sampling procedure

123 MAFF

Before establishing guidelines

Large variety but small sample number for one commodity

e.g. only 5 samples for one commodity

No attention to quality of analytical result

Poor requirement of internal quality control,

proficiency testing, precision/recovery data for the used method

Priority to low financial cost

124 MAFF

Change in design of surveillance in MAFF (1)

After establishing guidelines

Statistic based sample number

e.g. enough to obtain occurrence data on 95 percentile value with 95% confidence level

Use of validated method

Requirement of internal/external quality control

• method validation/verification data

• participation to proficiency testing program

• ISO/IEC 17025 accreditation

Priority to data quality

125 MAFF

Change in design of surveillance in MAFF (2)

Challenges for enhancing quality

assurance in Japan

Continued capacity building of risk assessors/

managers

Enhancement of quality assurance of official

and private laboratories and research institutes

Collaboration for analytical method validation

Supply of standard reagents and certified

reference materials

126 MAFF

Thank you for your attention!

127 MAFF

quality assurance of monitoring/surveillance data -...

Documents