instrument_adjumtment_policies_(reese)_06

1

Instrument Adjustment Policies

July 21, 2015

Instrument Adjustment Polices

NCSL International Workshop & SymposiumJuly 21st – 23rd, 2015 Grapevine TXSession 3C

Author/Presenter: Paul ReeseMetrology Sr. Principle EngineerMedical Products – PSE Operations

2015 NCSL International Workshop & Symposium – Grapevine TX (P. Reese)July 21, 2015 2


QUESTION: Is adjustment during calibration beneficial?

• Is “Accuracy” improved?

− Short Term?

− Long Term?

• Is End-Of-Period-Reliability (EOPR) improved?

• Can adjustment decrease accuracy or EOPR?

• When is adjustment required?

• When is adjustment optional?

• Is there an optimal adjustment threshold?

• Who decides when to adjust?

It is a deceivingly simple question. “Yes!” seems to be the intuitive response.

A definitive answer has proven elusive. Many opinions exists, but little data.



CALIBRATION: “Operation that, under specified conditions, in a first step, establishes a relation between the quantity values with measurement uncertainties provided by measurement standards and corresponding indications with associated measurement uncertainties and, in a second step, uses this information to establish a relation for obtaining a measurement result from an indication… NOTE 2: Calibration should not be confused with adjustment of a measuring system, often mistakenly called “self-calibration”, nor with verification of calibration”.

ADJUSTMENT: “Set of operations carried out on a measuring system so that it provides prescribed indications corresponding to given values of a quantity to be measured… NOTE 2: Adjustment of a measuring system should not be confused with calibration, which is a prerequisite for adjustment”.

VERIFICATION: “Provision of objective evidence that a given item fulfils specified requirements… EXAMPLE 2: Confirmation that performance properties or legal requirements of a measuring system are achieved…NOTE 3: The specified requirements may be, e.g. that a manufacturer's specifications are met… NOTE 5: Verification should not be confused with calibration”.

VIM Definitions: (The Fundamentals)



ISO/IEC 17025:2005(E) (R2010)“General Requirements for the Competence of Testing and Calibration Laboratories”

ANSI/NCSL Z540.3-2006 (R2013)“Requirements for the Calibration of Measuring and Test Equipment”

National and international standards for calibration do not requirethe routine adjustment of “In-Tolerance” items during calibration.

But is adjustment during calibration beneficial?



1. Adjust always

2. Adjust only if Out-Of-Tolerance (OOT)

3. Adjust with discretion when In-Tolerance and always when OOT

Three adjustment policies are practiced in calibration labs:

NCSLI RP-1 (terminology)

“Establishment and Adjustment of Calibration Intervals”

• Renew-always

• Renew-if-failed

• Renew-as-needed



Adjusting a lower accuracy Unit Under Test (UUT)…

to match a higher accuracy Standard (STD)…

generally improves the accuracy of the UUT, especially where high Test Uncertainty Ratio (TUR) exists.

…at least for the short-term.

But again, is adjustment during calibration beneficial?



Consider purely random instrument behavior (error) following a normal distribution...This is the performance if left unadjusted.

It is “In-Tolerance” 95 % of the time.

Variation is “common-cause”.

No systematic drift.



It is “In-Tolerance” 95 % of the time.

Variation is “common-cause”.

No systematic drift.

Consider purely random instrument behavior (error) following a normal distribution...This is the performance if left unadjusted.



The so-called “Deming funnel” experiment is widely used to demonstrate the futility of attempting to correct for random or “common-cause” errors.



If we try to improve the accuracy by adjusting the position of the funnel after each drop, we succumb to “Rule #2” temptation of the Deming funnel (tampering).

Let’s compare the two approaches by simulating large numbers of trial experiments (Monte Carlo style).

“I can fix this error.”

“Let me try that again.”

“Hmm... Maybe not.”



Rule #1: Never Adjust Rule #2: Always Adjust

95 %In-Tolerance (Reliability)

Std Dev = σ0 Std Dev = σ0

83.4%In-Tolerance (Reliability)

See B. Weiss, “Does Calibration Adjustment Improve Measurement Integrity?”1991 NCSL Proceedings

Also see “Analysis of Calibration Renewal Policies” by J. Larsen & K. Chhongvan. Published in Conference Proceedings of the 1994 Test & Calibration Symposium by the American Society of Naval Engineers (hosted by Naval Warfare Assessment Div)

Also see “Calibration Requirements Analysis System” by Dr. Howard Castrup.

1989 NCSL Proceedings



Rule #1: Never Adjust

95 %In-Tolerance (Reliability)

Std Dev = σ0

• But “never” means never adjust, even for OOT.• Not many calibration labs can ignore OOT’s.• Adjustment of OOT conditions is mandatory.

NCSLI RP-1“Establishment and Adjustment of Calibration Intervals” calls this:

“…a logical predicament”



“If we can convince ourselves that adjustment of in-tolerance attributes should not be made, how then to convince ourselves that adjustment of out-of-tolerance attributes is somehow beneficial?

For instance, if we conclude that attribute fluctuations are random, what is the point of adjusting at all?

What is special about attribute values that cross over a completely arbitrary line called a tolerance limit?

Does traversing this line transform them into variables that can be controlled systematically?

Obviously not.”

NCSLI RP-1

the “logical predicament”



“However, if a systematic mean value change mechanism, such as monotonic drift, is introduced into the model, the result can be quite different [Weiss-Castrup model]…

By experimenting with different combinations of values for drift rate and extent of attribute fluctuations… it becomes apparent that the decision to adjust or not adjust depends on whether changes in attribute values are predominately random or systematic”.

• “If random fluctuation is the dominating mechanism for attribute value changes over time, then the benefit of periodic adjustment is minimal”.

• “If drift or other systematic change is the dominating mechanism for attribute value changes over time, then the benefit of periodic adjustment is high.”

NCSLI RP-1



Almost all instruments exhibit both drift and random fluctuations (error).

We want to know how far “off from center”the funnel is (bias). Then we will know when to adjust is prudent (move the funnel).

hinders our efforts, but repeated measurements can help.

Instrument behavior is not typically limited only to random errors.

can be long-term (1/f).



If the mean of several repeated measurements shows a bias (offset), an adjustment can eliminate the observed bias.

… at least for the “short-term”.

Are these errors:

• “Systematic “Bias” …or

• “random” errors?

They differ only in the time period between observations (better repeatability than reproducibility)

…what about a 1 year cal interval?

If adjustments would have been made, short-term accuracy (time-of-test) would have been improved. But long term performance would have been degraded (increased spread).

There’s no monotonic drift. The temptation to adjust is great, but may not be prudent.

Let’s look at 5 cals over 5 years…(4 repeated measurements per cal)



Photo: Robert Moore (CC BY 2.0) Photo: Jinjian Liang (CC BY-NC-ND 2.0) Photo: Liam Murphy (CC BY-SA_2.0)

When does random variation become systematic bias?

Photo: Benefactor123 (CC BY 3.0)

It depends on the “sampling parameters” (e.g. time, distance, etc.)

To ants, one ripple in the road on the left is systematic bias.

https://www.flickr.com/photos/8860560@N02/4656156084

https://creativecommons.org/licenses/by/2.0/

https://www.flickr.com/photos/liangjinjian/1578236308/

https://creativecommons.org/licenses/by-nc-nd/2.0/

http://www.geograph.ie/photo/530871

http://creativecommons.org/licenses/by-sa/2.0/

https://commons.wikimedia.org/wiki/File:Corrugated_Road_-_Fremont_California.jpg

http://creativecommons.org/licenses/by/3.0/



Long-term random fluctuations or noise(i.e. noise with a long 1/f period = “meandering”)may appear to be actual bias errors when the observation period (e.g. during calibration) is short.

What appears as random over the long-term may be a systematic bias in the short-term.

See “The Myth of the Random Error” by Dr. Henrik Nielsen,Proceedings of the 1998 NCSL Workshop and Symposium.

With the exception of quantum phenomena, all apparently random “common-cause” fluctuations may ultimately be attributable to systematic causes, whether we understand those causes or not. Multiple superimposed systematic causes can easily manifest in an apparently random behavioral effect.

The meaning of the terms random and systematic should ideally be qualified with its sampling parameters (time period, distance, etc.).



So what should we do?

• If random fluctuation is the dominating mechanism for attribute value changes over time, then the benefit of periodic adjustment is minimal.

• If drift or other systematic change is the dominating mechanism for attribute value changes over time, then the benefit of periodic adjustment is high.

How do we know how much behavior is random and how much is systematic (e.g. drift) ?

Control charts are an excellent tool…

but are impractical for all test points on all equipment (e.g. RF Spectrum Analyzers can have 900+ test points).

← This chart is for ONE parameter



USL

LSL

Calibration Events (Years) → …real-world calibration histories of general purpose TM&DE often contain physical adjustments, reflected in the data.

These adjustments must be mathematically removed prior to regression analysis.

Not only are control charts impractical in many cases, but a 2nd complication exists.

While control charts are appropriate for reference standards (tracked, rather than adjusted)...

2015 NCSL International Workshop & Symposium – Grapevine TX (P. Reese)July 21, 2015


Red Dashed Lines Indicate Physical/Electrical

instrument Adjustment

Sometimes, NO AJDUSTMENT is Performed

Blue Lines Represent Instrument Behavior Between Calibrations

In reality, these are not straight lines; they

likely “meander about” between calibrations



Not Adjusted.No Translation

Necessary.

Remaining Points Translated Down

Similarly

Scale of Y-Axis Must Be

Extended

This now forms a theoretical data set, that represents the

pure instrument behavior, as if no physical adjustments had

ever been performed.

Anchor point -75 %(1st As-Found)

-87% – [(-13%)–(-75%)] = -148 %

-98% - [(-7%)–(-87%)] = -178 %

-98% - [(-7%)–(-87%) + (-13%)–(-75%)] = -240 %

-58% - [(-28%)–(-98%)] = -128%

-58% - [(-28%)–(-98%) + (-7%)–(-87%)] = -208%

-58% - [(-28%)–(-98%) + (-7%)–(-87%) + (-13%)–(-75%)]= -270 %

+10% - [(-3%)–(-58%)] = -45 %

+10% - [(-3%)–(-58%) + (-28%)–(-98%)] = -115 %

+10% - [(-3%)–(-58%) + (-28%)–(-98%) + (-7%)-(-87%)] = -195 %

+10% - [(-3%)–(-58%) + (-28%)–(-98%) + (-7%)-(-87%) + (-13%)–(-75%)] = -257 %

-80% - [(+10%)–(+10%) + (-3%)–(-58%)] = -141 %

-80% - [(+10%)–(+10%)] = -80 %

-80% - [(+10%)–(+10%) + (-3%)–(-58%) + (-28%)–(-98%)] = -211 %

-80% - [(+10%)–(+10%) + (-3%)–(-58%) + (-28%)–(-98%) + (-7%)–(-87%)] = -291 %

We want to mathematically remove the effects of all physical adjustments to the instrument. To do this, we subtract-out the cumulative effects of all previous adjustments for each “As-Found” data point.



Scale of Y-Axis Extended to -700 %

of Spec

Drift (Slope) ofRegression Line =

-41.5 % of Spec per Cal Interval

A linear regression can now be performed to estimate the instrument's inherent drift rate. All physical adjustments have been mathematically removed from this data set.



What about only adjusting if the error exceeds some threshold… ?

NCSLI RP-10 (2004), Section 9.7

Establishment and Operation of an Electrical Utility Metrology Laboratory

“Adjustments should be made (if applicable) or corrective action taken whenever the result of measurement consumes 80 % or more of the calibration tolerance…”

(x % of specification)

Maintaining control charts and historical tracking of all test points on all instruments is not practical for many laboratories.

Are there any “Rules of Thumb” that can help?



A Monte Carlo Model is presented for instrument behavior (drift + random).Accommodates prescribed “Adjustment Thresholds”.



The Monte Carlo model has an adjustableallowance for drift and random behavior.

The model also simulates(from 0 % to 100 % of Spec)adjustment thresholds.

0 % Adj Threshold = “Always Adjust”100 % Adj Threshold = “Adjust only if OOT”



This paper, as published in the NCSLI Proceedings, contains a detailed description of the first five iterations of this Monte Carlo simulation(Appendix A).

Parameters used in the example are:

Drift (µ) = 10 % of Spec/intervalRandom (σ) = 50.04 % of SpecAdj Threshold = 80 % of Spec

Tallies the number of OOT’s and computes EOPR after thousands of iterations (106)



Assumption of the Monte Carlo parameters used in this model:

Specifications are intended to represent relatively high confidence (i.e. 95 %)

If behavior is 100 % random (& takes up all the Spec), no allowance for drift is possible.If behavior is 100 % drift (& takes up all the Spec), no allowance for random is possible.

As more bias from drift is allotted, the random allowance must be decreased in order to maintain 95 % containment of errors.

Drift and random variation are modeled as complementary to one another in this investigation.



As the drift parameter (µ) in the model increases (more systematic bias),the random parameter (σ) is decreased to maintain 95 % containment probability.



• Drift (µ) = 1 %• Random (σ) = 51.01 %• Adj Threshold = 0 % Compute EOPR (106 Iterations)

Increment Adjust Threshold to 1 %Repeat 106 Iterations. Compute EOPR.



...Increment Adjust Threshold to 100 %Repeat 106 Iterations. Compute EOPR.

Now increment Drift (µ) to 2 %(Random σ is decreased to 50.98 %)and repeat entire process.

2 %, then 3 %, then 4%, etc.Random (σ) error is decreased, as drift increases.

~1010 Monte Carlo Simulations Total(brute force approach)

1 %2 %3 %100 %50.98 %



Drift →

Adj Threshold →

EO

PR

→



Dri

ft →

Adj Threshold →

EO

PR

→Purely random behavior (no drift),Adj Threshold = 0 % (always adjust).

The model yields 83.4% EOPR, just as predicted by the Deming funnel.

Reveals the effect of “tampering”, attempting to correct for “noise” or common-cause variation…“Over-Adjustment”.

Interestingly, adjustment thresholds of 100 % (adjust only if OOT) also yield the same depressed EOPR (83.4 %). EOPR is constant over this range.



Only after the adjustment threshold greatly exceeds 100 % does EOPR achieve 95 % EOPR.

With very large adjustment thresholds (>>100 %), no adjustments are essentially ever made (Deming Funnel Rule #1 = No Adjust). This results in the expected improvement in EOPR (95 %).

Monte Carlo Simulation is highly sensitive to very improbable events.

Cal Labs generally cannot employ adjustment thresholds >100 %

Achieves Deming’s “No Adjust Rule” Performance



Selected surface contour curves presented for eight specificdrift & random data pairs (percentages of specification).

Key Conclusions*:Increasing adjustment thresholds never results in improved EOPR(barring thresholds >>100 % of specification)

EOPR remains constant, or decreases, as adjustment thresholds are raised from0 % (always adjust) to 100 % (adjust only if OOT).

*Valid only under assumptions stated in published paper (see NCSLI Proceedings).



Key Conclusions*:All adjustments result in some amount of “tampering” where any random variation is present.

As a result, adjustments will add some entropy (noise) to the system, but they can also reduce significant bias (systematic error).

*Valid only under assumptions stated in published paper (see NCSLI Proceedings).




Key Cautions:These conclusions aren’t universaland may not apply to all instrumentbehavioral models under all conditions.

Just because more frequent adjustments(i.e. lower adjust thresholds) may result inhigher EOPR, doesn’t mean it’s always the mostprudent policy. Cost, compatibility with intervalanalysis systems, proprietary OEM adjustment software, interaction between Adj functions, etc. must be considered.


instrument_adjumtment_policies_(reese)_06

Documents

instrument adjustment

routine adjustment

adjustment optional

calibration of measuring

verification of calibration

calibration beneficial

calibration labs

optimal adjustment threshold