8/8/2019 Analyzers Accuracy

1/15

Description of Analyzerperformance

(Accuracy & Precision)

8/8/2019 Analyzers Accuracy

2/15

Accuracy & Precision

What do our customers expect from our analyzers?

How will they convince themselves?

0

5

10

15

20

25

30

35

40

45

0 5 10 15 20 25 30 35 40 45

Accepted Standard Laboratory test

FT-NIRA

nalyzerresult

Lab error

Analyzer error

Measure a single sample many times

8/8/2019 Analyzers Accuracy

3/15

Testing the performance with a single sample

0

5

10

15

20

25

30

35

40

45

0 5 10 15 20 25 30 35 40 45

Accepted Standard Laboratory test

F

T-NIRA

nalyzerresult

Lab error

Analyzer error

Error in repeated measurements= Repeatability

Below we see different results for the FT-NIR analyzer and the AcceptedStandard laboratory test

We can see that the analyzer agrees with the Accepted Standard

Laboratory test because both are centered at 25.

8/8/2019 Analyzers Accuracy

4/15

Accuracy

Common (old) Definition of accuracy Accuracy is measure of deviation from true value.

Who knows the true value!!

Replace Accuracy with Agreement (NIST directive)

Error in agreement between analyzer

result and result obtained with acceptedstandard reference method.

8/8/2019 Analyzers Accuracy

5/15

Agreement

How do we determine it?

Measure a sample or process product with FT-NIR analyzer and in

laboratory using accepted standard method

Compare the results

For one measurement result can be offset or error

Repeat procedure many times

Many = at least 20 times

20 comparisons using the same sample 20 comparisons tested by same operator on same day

20 comparisons with different samples on different days tested in (different)

lab by different operators

8/8/2019 Analyzers Accuracy

6/15

Agreement

Test Agreement between analyzer and Accepted Standard Laboratory test

with many samples

y = 0.999x + 0.014

20

22

24

26

28

30

32

34

36

38

20 22 24 26 28 30 32 34 36 38

Accepted Standard Laboratory test

FT-NIR

analyzer

8/8/2019 Analyzers Accuracy

7/15

Estimation of agreement error

0

5

10

15

20

25

30

35

40

45

0 5 10 15 20 25 30 35 40 45

Accepted Standard Laboratory test

FT-N

IRA

nalyzerresult

Combined error

The agreement error between analyzer and Accepted Standard Method is the vector

sum of the repeatability of each.

The agreement error is always greater than the repeatability of the the analyzer or theAccepted Standard Method

When repeatability of both are same, agreement= 1.41x error of Accepted Standard

Method

8/8/2019 Analyzers Accuracy

8/15

Summary on Agreement

(guaranteed) specification for agreement between our analyzer and the AcceptedStandard Method

We must know the repeatability of the Accepted Standard Method.

As estimated in the protocol

As used by the customer

For several operators, different samples, different labsetups etc.

It is best to obtain real up to date statistical data from the customer Use work sheet for determination of standard error of duplicate determinations

(Pooled standard deviation)

We must also know the repeatability of the analyzer.

Often much better than Accepted Standard Method. Rule of Thumb1: Check performance of customer lab carefully

Rule of Thumb2: Leave some margin in guaranteed spec.

Quote between 2 and 1.5 times the Standard error of duplicate determinations of

Accepted Standard Method.

8/8/2019 Analyzers Accuracy

9/15

Summary on Agreement (cont.)

Agreement specification is needed when product is tested with Accepted

Standard Method.

By Government inspector to comply with regulatory quality

By customer to provide custody transfer certification By customer to insure product in process quality

You can see with term agreement, we are not bound by Accepted

Standard Method.

We can establish agreement specification between any 2 methods of

analysis as long as we know their repeatability spec.

8/8/2019 Analyzers Accuracy

10/15

Repeatability and Reproducibility

Repeatability is specification of error for repeat determinations with allconditions the same

Formerly called precision

Reproducibility is specification of error for repeat determinations wheresome conditions may be different

Different operators

Different labs

Different Spectrometers

Our specification of absorbance reproducibility fromspectrometer to spectrometer gives us the possibility ofcalibration transfer

8/8/2019 Analyzers Accuracy

11/15

Summary on Repeatability & Reproducibility

Repeatability indicates how precise an analyzer can analyze theproduct

The repeatability error is commonly much smaller than the agreement

error In process monitoring it may be more important than agreement

It insures that the product is always the same to within a small error.

Reproducibility of our analyzers is important for customers who want toproduce the same product at different sites.

8/8/2019 Analyzers Accuracy

12/15

Agreement

OHvalue

40

45

50

55

60

65

70

75

80

85

40 45 50 55 60 65 70 75 80 85

Accepted Standard lab method

FT-NIR

analyzer

RMSD=0.7 (2 Factors)

8/8/2019 Analyzers Accuracy

13/15

Repeatability

OH value

70

71

72

73

74

75

76

77

78

79

80

70 71 72 73 74 75 76 77 78 79 80

Actual

Predicted

cal samples

Validation

Linear (cal samples)

Repeatability RMSD 0.05

8/8/2019 Analyzers Accuracy

14/15

Repeatability

control chart 10 weeks

y = 0.0002x + 57.202

56

56.2

56.4

56.6

56.8

57

57.2

57.4

57.6

57.8

58

0 20 40 60 80 100 120 140

days

O

H

value

RMSD= 0.05

8/8/2019 Analyzers Accuracy

15/15

Reproducibility

System to System Reproducibility

55

56

57

58

59

60

1 2 3 4 5 6

FTLA2000-160 unit

OH

value

STDEV (from unit to unit) = 0.11