jcgm/wg1 template for reporting the comments on supplement 1 · transition between old and new...

JCGM-WG1-SC5-N16-08b Date: 9 September 2016

of 90

JCGM-WG1 template for reporting the comments on JCGM 100:201X CD JCGM-WG1 responses to National Metrology Institutes (NMIs): Ural Research Institute for Metrology (UNIIM), Russian Federation National Physical Laboratory (NPL), UK Central Office of Measures (GUM), Poland Czech Metrology Institute (CMI), Czech Republic Centro Español de Metrología (CEM), Spain Measurement Standards Laboratory of New Zealand (MSL), New Zealand Dutch National Metrology Institute (VSL), Netherlands D.I. Mendeleyev Institute for Metrology (VNIIM), Russian Federation Korea Research Institute of Standards and Science (KRISS), Republic of Korea Centro Nacional de Metrología (CENAM), Mexico Physikalisch-Technische Bundesanstalt (PTB), Germany Laboratoire National de Métrologie et d'Essais (LNE), France Federal Institute of Metrology (METAS), Switzerland National Institute of Metrology (NIM), China Instituto Nacional de Tecnología Industrial (INTI), Argentina National Metrology Institute of Japan (NMIJ), Japan National Scientific Centre "Institute of Metrology" (NSC IM), Ukraine National Measurement Institute (NMIA), Australia National Institute of Metrology, Quality and Technology (INMETRO), Brazil

Please, fill in the Table sequentially according to the text

M.O. or NMI

Serial item No.

(Sub) clause

(e.g. 4.1)

Paragraph No. / Line

No. / Figure /Table/Note

Justification for change Proposed change Response of JCGM-WG1

1. NPL 4 General

The JCGM should consider a ‘four page summary GUM’ to be published alongside the new document to engage stakeholders better.

Consider the production of a four page ‘summary GUM’ to be published together with JCGM 100 and JCGM 110

Under consideration.


of 90

2. NPL 5 General

There are a number of general points, not specifically related to the text of the document, which the JCGM should consider in order to maximise the impact and accessibility of the new GUM and smooth the transition between old and new documents.

• The JCGM considers any impact on the formatting and structure of the KCDB which may be required in order to properly report uncertainties associated with CMCs. • The JCGM consider the impact of the changes on other standards (for example, ISO 5725 and ISO 21748) and whether any plans for wider harmonisation are required. • The JCGM should assist with the education of all communities about any proposed change by publishing appropriate articles in relevant trade and learned journals. • The JCGM should encourage the publication in the peer-reviewed literature of full uncertainty budgets using the new GUM approach. • The JCGM should seek increased feedback from end users of the GUM, for instance accredited laboratories and instrument suppliers, to ensure they have been directly consulted. • The JCGM and NMIs should provide assistance to accreditation agencies in reworking end users uncertainty documentation (such as UKAS M3003, and EA-4/02). • The JCGM provide clear information about proposed timescales for adoption of the new GUM and guidelines for arrangements during transition:

Thank you.


of 90

in particular for CIPM Key Comparisons.

3. CMI 10 General

Out of practical reasons, it would be advisable to cover here the controversial concept of inclusion of the contribution to the combined uncertainty of the unit under test.

Accepted Some material will be considered.

4. CMI 11 General

Again, out of practical reasons, for instruments in use the concept of uncertainty growth should be added, e.g. along the lines of Chapter 9 of NASA Handbook NASA-HDBK-8739.19-3.

As above.

5. CMI 12 General

The laboratory standard ISO 17025 is now under revision and one of the proposals is to integrate in it concepts from ILAC documents of traceability and uncertainties. Therefore, the development in both documents, GUM and ISO 17025, should be closely pursued to arrange for a proper alignment between them.

Agreed in principle, but JCGM-WG1 has no jurisdiction over the ISO committee responsible for 17025.

6. CMI 13 General ISO Guide

35

In certification experiments to assign a value to a RM the Guide stipulates that the component of uncertainty associated with this experiment can be taken as an average of individual contributions from laboratories participating in the experiment. Here the sample standard deviation of the arithmetic mean is used – if a randomly chosen RM is then used subsequently by a user to calibrate his or her instrumentation should not be the sample standard deviation used instead ?

Not agreed since outside scope.

7. CEM general

Some information regarding the way to estimate the uncertainty when corrections from a calibration curve are not applied (similar to F.2.4.5 from JCGM

Under consideration by JCGM-WG1.


of 90

100:2008) should be included in the new version because it is extremely useful.

8. CEM general

We would like to express our concern regarding the impact of the implementation of this new GUM in the accredited laboratories and industry in general. The implementation of the GUM in our country took a long time (several years) and only after hundreds of hours of training it was finally achieved. It seems that this new GUM involves the use of specialized software to perform the calculations and we fear that for many small and medium size laboratories the procedure to evaluate uncertainties will be transformed into a "black box" entailing a lack of knowledge about the evaluation itself, with all the inherent dangers.

Black-box software pervades many metrology disciplines. This situation has to be handled.

9. VSL 1 Entire document

We welcome the improved consistency in evaluating standard uncertainty and the clear structure of the document. We feel that those areas where substantial changes have taken place (in particular in type A evaluations), more attention should be paid to exemplifying the new approaches

Provide more examples for evaluating standard uncertainties, including parameters obtained from regression and analysis of variance.

Agreed. Many examples will be included in the evolving JCGM 110.


We think that the document should be better supported by worked examples and references to easy-to-understand literature to enable non-mathematicians to work with the document.

Elaborate the suite of examples contained in JCGM 110; add references to statistical text books, open literature on key concepts (e.g., clauses 4 and 5)

Agreed. Many examples will be included in the evolving JCGM 110. JCGM-WG1 to discuss suitable literature.


of 90


We recommend working on a process to introduce the new GUM, including a reasonable time frame for making adaptations to processes, procedures, certificates and CMCs

Develop a document outlining a smooth implementation of the changes inflicted by the new GUM

Agreed. In hand by JCGM-WG1.


What is the economical impact of a new GUM? Will more consistent and reliable uncertainty estimates outweigh the additional effort required for adapting and using the new GUM?

Consider economical impact before introducing new GUM



Calibration, measurement and testing laboratories may find the new GUM too complicated. In particular the requirement to validate the use of the law of propagation of uncertainty may be a real burden for many laboratories and industries.

Consider expected educational level of end-users before introducing new GUM.



Guidance, such as in annex F.2.4.5 for dealing with results not corrected for bias is missing

Consider taking over F.2.4.5 of JCGM 100 in the new GUM.

Agreed. Under consideration by JCGM-WG1.

15. CENAM 1 general general

The metrology community using the GUM encompasses more than 100 NMIs and Dis, an estimated of 1000 calibration and testing laboratories for each NMI and DI, servicing 10 industries each, so amounting roughly to 1 million users. This figure still has to be multiplied by a factor of at least 3 to get an estimation of the type of calibrations and tests where a measurement uncertainty (MU) value is needed. Along two decades this community has been working on learning, assimilating and applying the GUM1 approach, and, even recognizing that the MU values could not be

Include a section identifying those situations where the results of GUM2 are significantly different from those of GUM1.



of 90

so accurate, they have been useful for the intended purposes, at least no notices have been heard against. Also, in many cases, the GUM1 results have been validated by Monte Carlo techniques. It is recognized that the proposed GUM2 adds up “metrological rigour” and that it incorporates some “advancements in the formulation and solution of problems involving measurement uncertainty”., and that brings in “more sophisticated techniques, potentially difficult for many users”. Thus, it seems that a bridge to bring the results of the community efforts to alignment with a more solid framework is in order. It would be senseless to erase the accumulated experience and to apply the new proposal at once; without any doubt a gradual implementation of GUM2 is advised. The perception is that GUM2 provides guidance to get more accurate MU estimations, still within hypothesis not far from those of GUM1. Nevertheless some difficulties in MU are not clearly resolved, like the related to regression analysis.

16. CENAM 2 general general

In chapter 4.2.5 of current version of GUM is mentioned how to deal with the variance due to the fitting of calibration curves (type A uncertainty). In example H3, thermometer calibration is showed how to calculate this contribution (formula H.13f). In the proposed document (new GUM), is mentioned that from Bayesian point of view, standard deviation resulting from type A is no longer an estimate of a standard deviation,

Add a paragraph explaining how to deal with this possible contribution due to the fitting of calibration curves or give a reference to the adequate supplement.

Accepted.


of 90

but a parameter of a state-of-knowledge PDF. There is not mentioned how to calculated the estimation of the standard deviation of the PDF due to the contribution of fitting in curves.

17. CENAM 42 general

In most of the measurement fields at primary levels of measurement, we anticipate no significant changes in the reported uncertainty. However, additional cost must be paid to analyzing and justifying the probability distributions of the input quantities.


18. CENAM 43 general

Costs can actually increase also for secondary laboratories and industry, due to lack of knowledge of the probability distributions, leading them to use conservative coverage intervals. It may be the case that they could exceed the control limits or fail acceptance tests more frequently due to their conservative expanded uncertainties.


19. PTB 2 General

Structure within sections: to increase acceptance of the revision, the order of clauses should be closer to the present GUM (e.g. Tab. 2).

Reorder clauses where appropriate.


20. PTB 3 General

It is not always clear, whether a specific text belongs to a note or to the main text.

Clearly separate the notes from the main text, e.g. by using another font or by indenting the notes.

Notes are already in a smaller font.

21. PTB 4 General

For the orientation in the document, it would be very helpful to have headlines even for all sub-clauses.

Insert headlines to all sub-clauses, e.g. for “10.4.2.3” the headline “Product of input quantities” and for “11.4.1” the headline “Gaussian distribution” would be appropriate.

. Doing so would render the document unwieldy and increase the departure from the current GUM.


of 90

22. LNE 1 Document & 1.2

It may be useful to better outline the need of the changes from all points of view (ex. scientific rigour, accuracy of the measured value and the measurement uncertainty but also from the users point of view: NMIs and industry)

Introduce a new paragraph like “Why a new version of GUM?”


23. LNE 2 Document & 1.2. d

“the concept of an uncertainty having effective degrees of freedom is no longer needed” but one can find degrees of freedom in different clauses of this document. Then the proposal has an internal contradiction on the use of degrees of freedom. Furthermore the statements for the GUM Annex G regarding the degrees of freedom are not specific to frequentist statistics and are relevant both in Bayesian and non-Bayesian contexts

Delete the sentence in 1.2. d and keep in the proposal the materials provided by GUM Annex G regarding DoF

Agreed in part.. JCGM-WG1 to review this aspect. Effective degrees of freedom is retained regarding state-of-knowledge distribution based on, say, information on calibration certificates.

24. LNE 3 Document & 1.2 e

The Bayesian approach proposed seems questionable, mainly because it does not take into account some essential concepts in metrology (in particular regarding radioactivity), such as uncertainty of definition, design or construction of the measurand: “There is no consideration of definitional (or intrinsic) uncertainty”

Propose guidelines to take into account the definitional uncertainty or justify clearly why it is not covered by this document


25. LNE 4 Document & 8.1.2

The main issue of metrology is often to establish this functional relationship, to identify all influence parameters and their possible correlation. And often the main cause of uncertainty originates in the use of an incorrect functional relationship.

General revision of the text after an in-depth study of the concept of measurand and the ways to take into account intrinsic uncertainties due to its definition. The study of the possibility to use robust

Will be considered in JCGM 103.


of 90

estimators of the variance with no prior, in the case of repeated measurements, must be explored. The total rejection of frequentist (or minimum hypothesis) approaches must be justified.


“This Guide provides methods for evaluating and reporting measurement uncertainty that apply from the shop floor to fundamental research” We experience a lot of difficulties in making users, customers and even students understand and adopt the rules of the present GUM. The document should be clearly understandable and simple to use, and not aimed only at metrology experts. The cost / benefit ratio of this revision should be clearly evaluated by all parties (metrologists, calibration laboratories, end users, industrials, accreditation bodies) before adoption.

General revision of the text after a reflexion on the cost/benefit ratio of the proposed changes, mostly motivated by a Bayesian approach of probability. Precise evaluation of the practical problems encountered in the application of the original GUM by various users, including NMIs.



The Bayesian view of probability adopted in this guide does not consider the prior knowledge about the measurand. The Bayes’ rule applies here only to input quantities and not to the measurand without any justification, which is not consistent in a Bayesian context.

Provide a consistent Bayesian treatment of the measurement uncertainty

JCGM-WG1 plans a separate Bayesian document.

28. LNE 7

Document & 3.5

The estimation of kp will involve huge financial consequences for routine laboratories and also consequences on the final result when using conservative coverage intervals. Laboratories should then verify and update all their kp.

Keep the more simple and previous guidelines to compute kp or simply recommend to use GUM S1



of 90

If the measurand cannot be taken as Gaussian and without any additional assumption, then kp= 4.7, which involves an uncertainty multiplied by more than 2!!! In some cases it will invalid some measurements and some existing accreditations, lead to different conclusions in conformity assessment and to review all CMCs!

29. LNE 8 Document

Evaluating the uncertainty budget is a systematic and complementary task to the uncertainty evaluation. Simple calculations enable the user to identify the input quantities that contribute the most to the variance of the output quantity, which is the most valuable information. For example, in order to improve its measurement process one would save time and money by quickly identifying his main contributory sources of uncertainty

Add a new Clause 13 called “Evaluating the uncertainty budget” with sections : 13.1 General 13.2 Computation of the uncertainty contributions (based on the sensitivity coefficients) 13.3 Limits (Correlated input quantities and non-linear measurement model)


30. LNE 9 Document

The term “variance“ is avoided in the document. For example when there are formulae to compute variance (3), (31), (35), “standard uncertainty“ is preferred instead. This is very confusing since standard uncertainties and covariances are placed on the same plan and may lead to mistakes while combining standard uncertainties and covariances

Explain that, for calculation, variances are needed because one cannot combine standard uncertainties. In ordert to report the dispersion in the same unit as the position parameter, to take the square root and report the standard uncertainty is necessary

Similar in this regard to GUM-S1.

31. METAS 0 general We appreciate the great work of the Joint

Committee for the revision of the GUM. The new GUM is scientifically sound, better readable and more concise than the actual version and in some cases also somewhat easier to apply.

Excellent! 1. There is a penalty on obtaining greater reliability of results – see


of 90

Nevertheless, we have severe concerns to its introduction in the present form. The reasons for this are: 1. The current version of the GUM is now

finally, after many years of training and advocacy, well introduced in the user community. We have no signs that the uncertainty concept introduced by this document has led to severe inconsistencies. The new GUM may be scientifically sounder than the old one, but we cannot see that it will lead to a major improvement in the evaluation of uncertainties in most practical applications. On the other hand, the workload for the NMIs, the accredited laboratories and industry to implement the new GUM will be very important. We doubt that the necessary changes will be implemented because a reasonable relation between costs and benefit may not be perceived.

2. The new GUM gives the impression that linearization of the measurement model and the assumption that the output PDF is Gaussian is only correct in exceptional cases. In practice, however, these approximations work well in the vast majority of cases. This method even has the advantage that it immediately provides an uncertainty budget (through the sensitivity coefficients). Something that’s

ISO 17025 – it is necessary to carry out a proper evaluation free of unverified assumptions. If the user wants to assume normality, he can, but should state he is making this assumption. An uncertainty budget applies only to input quantities that are independent and a measurement model that can reliably be linearized. In the proposed revised GUM we are trying to remove assumptions and the “don’t know” element as far as


of 90

much more elaborate to achieve by Monte Carlo. Practical difficulties in measurement uncertainty evaluation are much more related to the modelling of the measurement process, which is always just an approximation to reality, and the identification of all influences. Compared to that, the influences due to nonlinearities in the measurement equation and deviations from the Gaussian form of the output-PDF are usually negligible. And any smaller errors caused by these latter effects are typically covered because the careful metrologist is aware that his/her model is never perfect and accordingly assigns conservatively larger uncertainties. The new guide should be more pragmatic in that respect.

3. The effect of the new GUM is, that in the most general case, the coverage factor of k = 2 will have to be increased to k = 3 (the hurdle to assume that a PDF is Gaussian is much higher than before). This will not be acceptable in industry and will have large consequences in the assessment of conformity with manufacturing tolerances. On the other hand, the practice to use k = 2 for a 95% confidence interval has proven to work reasonably well, although scientifically not being entirely correct.

4. No practical guidance is given to support users of the new GUM on how to manage

possible. A modelling document is well under development. 3. Under consideration. 4. Under consideration by JCGM-WG1.


of 90

the transition.

32. INTI General comment

Does the bayesian approach enlarge the uncertainties? How much? How is the impact caused by the application of the factor ((n-1)/(n-3))1/2 in Type A evaluations? Perhaps, it is not relevant for most cases. But a broad study, comparing both methodologies for many real cases would be a good idea, before to put in force the new one. It can clear up doubts or concerns that could exist among the users, generating resistances. These studies can be performed by volunteer NMIs, managed by the JCGM

To produce documents comparing the final uncertainty by both methodologies (old and new GUM) for a series of calibration and measurements procedures


33. INTI

General comment

The comprehension of the new GUM Despite the GUM 2008 is based on some confusing theoretical basis (a mixture of bayesian and frequentist approaches), it was written caring that the different concepts were simply and straightforwardly expressed. For instance, the Chapter 8 gives a very brief summary of the procedure, for effortless comprehension. Then, in the subsequent years, other documents were published, in order to simplify the understanding of the Guide and focusing in specific applications (e.g., NIST TN 1297 in USA, EA-4/02 in Europe, the EURACHEM/CITAC guide for Chemistry, etc). All this effort contributed to extend the application of the GUM around the world. During the last two decades INTI spent a considerable amount of work in training and consulting on the GUM methodology in Argentina as well in other SIM countries.

To include a chapter with summary of the procedure(s) stated To promote the development of other documents in order to facilitate the understanding

A summary procedure is provided. The Bayesian content is minimally changed from the current GUM (only regarding Type A evaluation when repeated observations are available). JCGM-WG1 intends to de-emphasize Bayesian aspects further than in the proposed


of 90

Many courses and assistances were given to NMIs, to universities, to secondary laboratories and to the industry in general. From that experience, we can say that the statistical complexity was not an obstacle to comprehend the Guide and to apply it correctly (whereas, the lack of specific metrological knowledge usually was an impediment). To solve most of the real problems and cases presented to the users, only some basic statistical/probabilistic concepts were sufficient: standard deviations, degrees of freedom, normal and rectangular pdfs, coverage probabilities, and no much more. So, many users without statistical training became able to deal with uncertainty evaluations. Therefore, a broadly extended diffusion of the GUM was possible at the “ground” level, beyond NMIs (calibration and testing labs, industry, commerce, etc). In the new version, knowledge of deep statistical concepts (e.g., Bayesian analysis) is not really needed to be able to apply the methodology, similarly to as in the old one. However, some paragraphs, seem to be harder. So, this issue has to be considered to avoid some resistances that can appear. This fact can be a risk to the universality of the GUM scope in practice. A summary with a straightforward application is missing and could be necessary (similar as the Chapter 8 in GUM 2008)

revised GUM. JCGM-WG1 believes the state of the art of uncertainty evaluation should not stand still, just as in other areas of science. Further documents are planned or under develoment: modelling, Bayes, …

34. INTI General

To assure continuity between versions We recommend to include JCGM will


of 90

comment references to the corresponding paragraphs in the GUM:2008 when possible For instance: 7.2 Error, effect and correction (3.2. in JCGM100:2008)

consider issuing cross-reference.

35. NMIJ 1 Whole document

We appreciate the efforts made by the JCGM WG1 to develop an improved guide for uncertainty. While we agree that this draft will bring improved statistical consistency in the GUM, we also have to consider that, after more than twenty years since its first publication, the current GUM is already integrated into many social systems such as the national/international traceability systems, laboratory accreditation systems, and ISO/IEC standards. We have a concern that revision of the GUM in a short period of time, i.e. in a year or two, mainly from an academic viewpoint may have serious undesirable impacts on these systems. If the GUM is revised without deliberate consideration of extra cost and labor needed for reconstruction of such systems, respect toward the GUM cultivated so far among metrologists, engineers, and scientists would be lost, and further spread of the concept of uncertainty might be hindered. We also have fear that this draft might not cover the whole situations to which the current GUM has been successfully applied. Such situations include the use of ANOVA methods, "Type A" evaluation based on repeated measurements less than 3, and cases in which definitional uncertainties are not negligible. If the new methodology advocated in this draft is found not applicable to certain situations to which

Maintain the current GUM as JCGM 100 for now, and publish the draft guide as another document such as a supplementary guide to the GUM. Allow sufficient time to analyze what impacts the revision will have on the current metrological and other social systems, and decide whether and when the current GUM be replaced by the draft on the basis of this analysis.



of 90

the current methodology has been applied, there should certainly be a serious confusion and disappointment with the new one among users of the GUM. We therefore think that the current GUM should not be revised for now, and should be maintained as JCGM 100. Instead, it is desirable to publish this draft as another document such as a supplement to the GUM. In this way, we can take time to analyze the impacts that the GUM revision may have on the current social systems, while providing the opportunity to use the new methodology to those who wish to use it. Final decision of whether and when the current GUM be replaced by the revised guide should be made on the basis of this analysis.

36. NMIJ 2 Whole document

This draft and the current GUM give, in general, different estimates of uncertainty. It is desirable to provide explanation on how and when these two versions of the GUM give markedly different results. Also, we have to take special care when measurement uncertainties are related to safety issues.

Add an independent clause or an annex explaining how and when the two versions of the GUM give markedly different results. Also, add a notice that these differences may raise safety issues.

Under consideration by JCGM-WG1. Revised text will be backed up by JCGM 110 examples.

37. NMIA 1 All All

See email from NMIA dated 9 April 2015

38. LNE 10 Many places in the document

The expression “significant” decimal digits” used throughout the text is confusing and non-standard. In rounding numbers, one usually refers to “significant digits”, or “decimal significant digits” or “significant figures”, or “decimal places”

Use only “significant digit” If binary digits were considered, the conclusions would generally be different.

39. LNE 11 Title

Why do you change the title ? Add the sensitivity analysis in Under


of 90

As it is, there is no more “expression” of uncertainty, the “use and management ” of uncertainty” could be developed and specially the sensitivity to input values !

the document consideration by JCGM-WG1 . Sensitivity coefficient are in 10.4.1.1.

40. LNE 12 Title

As in JCGM 200 with (VIM), it’s very important to keep the acronym (GUM) in the title of the document

Title + (GUM) Under consideration by JCGM-WG1.

41. LNE 13 Title

If the title of this document is kept, the title of JCGM 104 should be modified accordingly.

Change Title of JCGM 104 in accordance with the title of this document


42. VSL 6 title

The current banner “Guide to the expression of uncertainty in measurement” covers better the contents of the document than the proposed title; the latter can be read as if it is a document giving guidance on how to introduce uncertainty in measurement

Revert to the title of the current GUM


43. PTB 1 Title

The title of the 1993/1995/2008 editions should be kept.

Use the former title: “Guide to the expression of uncertainty in measurement”


44. MSL 1 Foreword

P1/L4 Unnecessary text Remove “(specific fie)” Agreed.

45. VSL 7 Foreword

Various minor typing errors (e.g., specific fie, n(ILAC), n(IEC))

Please correct Agreed.

46. VSL 8 Foreword

“VIM” is undefined undefined Please give full name of the

VIM3 Agreed.

47. LNE 14 Foreword

Page V Line No4

Specify VIM after its designation …and the “International vocabulary of basic and general terms in metrology” (VIM). The JCGM…

Agreed.


of 90

48. VSL 9 Foreword

Link to form not clickable in paper form Provide full HTML-link or

appendix for submitting comments

Agreed.

49. VSL 10 Foreword

'Introduction' undefined capitalize 'General Introduction'

after JCGM 104:2009 Agreed.

50. VSL 11 Foreword

Table 1 Overview of documents is incomplete Add JCGM100 to the table Agreed.

51. CENAM 3 1.1.1

“Obligatory” seems a term overstating the requirement of MU reporting, save on regulatory framework. The Oxford Dictionary means “obligatory” as “required by a legal, moral, or other rule; compulsory”

Replace the term as appropriate.

Text will be amended.

52. INTI

1 1.1 1.1.1.

The sentence “When reporting a measurement result it is obligatory that it includes some quantitative evaluation of its quality” is not consistent with Note 2 of VIM 2.9: “If the measurement uncertainty is considered to be negligible for some purpose, the measurement result may be expressed as a single measured quantity value. In many fields, this is the common way of expressing a measurement result »

.”..it is “convenient” or “...it is advisable” or “it is obligatory in some contexts related to the scope of this document (see 2.1.)”

Text will be amended.

53. NSC “Institute

of Metrolo

gy”, Ukraine

1 1.1 1.1.3

In Clause 1.1.3 states that methods for evaluating of measurement uncertainty expressed in JCGM_100_201X_CD are internally consistent (identical results are obtained whether the problem is solved directly or decomposed into sub problems whose solutions are combined). However, our studies show that this statement is not obtained in all cases. Let us consider a measurement model as a sum of m input independed quantities:

Add the sentence in the end of 1.1.3: “In some cases the Bayesian approach is not be applied”.

JCGM-WG1 will consider.


of 90

1

m

jj

Y Х=

= ∑. (1)

In accordance to Clause 3.4 JCGM_100_201X_CD, the standard uncertainty of the measured value can be expressed in terms of the standard uncertainties of the input quantities as:

2

1( ) ( )

m

jj

u y u x=

= ∑. (2)

If estimates jX are obtained as a result of jn repeated indication values jix , where

1,2,..., ji n= , they will be equal, respectively:

1

1 jn

j jiij

x xn =

= ∑, (3)

and their standard uncertainties of type A ( )A ju x , in accordance to Clause 9.2.3

JCGM_100_201X_CD, will be equal to: 1

( )3

j jA j

j j

n su x

n n

−=

−, (4)

where js - the values of the sample standard deviation. Substitute (4) into (2) we get the standard uncertainty of type A of measurand:

2

1

1( )

3

mj j

Aj j j

n su y

n n=

− =

− ∑

. (5) For simplicity assume, that the evaluations of all input quantities are made with the same number of indication values jn n= and thus


of 90

for all evaluations of input quantities received

the same sample standard deviations js s=.

Then it follows from (5) we obtain: 1( )3A

n su y mn n

−=

− . (6) On the other hand, since the measured value for the model (1) will be:

1

m

jj

y x=

= ∑, (7)

and jx determined by the expression (3), that

1 1 1 1

1 1m n m n

ji jij i j i

y x m xn nm= = = =

= =

∑∑ ∑∑

. (8) In parentheses of the expression (8) - the arithmetic mean of all nm observed values jix with equal standard deviations. In this case the standard uncertainty of type A of the measurand must be equal to:

1 1( )3 3A

nm s nm su y m mnm nmnm n

− −= =

− − . (9) The difference in expressions (9) and (6) represent the same measurand, is obvious.

With increasing m the value of 13

nmnm

−− to

tends to 1, and then it follows from (9) we obtain:

( ) su y mn

=, (10)

that in 13

nn

−− times less than the estimate by


of 90

the formula (6). For n = 4 value 13

nn

−− is

1.73. Moreover, as follows from (6), the distribution law, attributed to the measurand is the t-distribution with degrees of freedom (n-1) regardless of the number of input values m. This assumption may lead to an incorrect evaluation of the expanded uncertainty. At the same time, from (9) follows that the distribution law attributed to the measurand (in the absence of the standard uncertainties of type B) is the t-distribution with degrees of freedom ( 1nm − ), and with increasing m, this distribution is infinitely tends to a Gaussian distribution. Thus, for the same initial data at different ways of processing technique JCGM_100_201X_CD give different estimates of uncertainty. The technique described in JCGM 100-2008, provides equal estimates of measurement uncertainty in both cases.

54. KRISS 1 1.1.4

Discrepancy for the purpose of the guide In GUM, the value of the measurand and true value are viewed as synonymous. In 1.1.4, this guidance is for the determination of an interval that contains the value of the measurand. Then, it means that ‘This guidance is for the determination of an interval that contains the true value’. In 5.1, uncertainty is defined as ‘parameter characterizing the dispersion of the values

Guidance is given on the determination of an interval that represents the dispersion of the values being attributed to a measurand with a given probability, intended as degree of belief.

JCGM-WG1 will consider textual changes for clarification.


of 90

being attributed to a quantity, based on the information. Throughout this guidance, it is understood that a value being attributed to a quantity, an (best) estimate of the measurand and measured (quantity) value are synonymous.

55. LNE 15 1.1.5

Appreciate to have bring forward this clause / Thank you.

56. LNE 16 1.2

Major changes are neither clearly explained, detailed nor complete

Explain more the changes that the new version of the GUM brings and their impact. Either all changes or that ones that impact the most the measurement result (ex. the coverage interval, the option to declare it, the justification of the adequacy of linearization etc)

Agreed. JCGM-WG1 to consider improved text.

57. INTI 1.2

The needs for the change of the Guide are not explained The GUM: 2008 is too well accepted and installed among users to be replaced without a convincing description of the justification for that. Basis and fundamentals should be developed

To include a summary of the reasons for the change


58. PTB 5 1.2 b)

Sentence badly readable Add the references to clauses “(see clause 3)” and “(see clause 8)” after “summarising the main procedure” and “measurement modelling”


59. METAS 1 1.2 c)

The omission of the term “combined” uncertainty is unnecessary and has nothing to do with the new approach. It has proven to be practical and would survive in practice anyway for a long time.

Keep the qualifier “combined” and the subscript “c” for “combined standard uncertainty”

Removed already in GUM-S1 and GUM-S2.

60. METAS 2 1.2 d)

Most readers will not be familiar with the Bayesian view of probably. The principles of this approach and the difference to the

Introduce the Bayesian approach in a few lines and explain the difference to the

Agreed. In addition, Bayesian


of 90

frequentist approach are not explained. frequentist approach. approach to be de-emphasized to gain wider readership.

61. CMI 1 1.2.

9.2.3

.

par. d) Here (and in a number of presentations of the authors in a preparatory stage) Bayesian view has been very much highlighted. The proposed treatment of the former type-A evaluation has by itself nothing common with Bayesian statistics (and scaled an shifted t-distribution is never mentioned in textbooks). Otherwise, the bayesian approach is reduced to a mere selection of the distribution at the discretion of the experimenter. I believe that it is confusing to both rank-and-file metrologist and a trained statistician and for the former this rather giant leap somewhere else in this type of evaluation should be more explained.

To add an annex to par. 9.2.3 where the use of the given t-distribution as an estimate of the distribution of the arithmetic mean is derived.

Reference will be added.

62. Inmetro 1 1.2 Item d)

Although the new GUM formally adopts the probabilistic Bayesian view, the guide does not address, in fact, the Bayesian approach for the evaluation of measurement uncertainty.

This fact could be better explained in order to be clear to the readers who have no prior knowledge of either the Guide or the Bayesian inference. In addition to make clear the main difference of Bayesian thinking about the assumptions of the parameters in statistical inference, suggesting references as [7, 9] would be interesting.

Agreed. Bayesian approach to be de-emphasized to gain wider readership.

63. GUM 1 1.2 d)

We suggest to omit the third sentence of the paragraph: “Thus, the concept of an uncertainty having effective degrees of freedom is no longer needed”, because the term “effective degrees of freedom” is used in Table 2 (first column, fifth line) associated

We suggest to use the sentences: “Thus, there will no longer be effective degrees of freedom attached to the output uncertainty. Accordingly, the Welch-Satterthwaite formula will no longer be needed”. Those



of 90

with the subclause 9.3.4, where also is used.

sentences were used by Walter Bich in Metrologia 51 (2014) S155-S158.

64. PTB 7 1.2d)

It is emphasized that the term effective degree of freedom is removed; nevertheless the term degree of freedom is still present in the Guide.

Add a note to 9.2.3 “The degrees of freedom are a parameter of the t- distribution, and they are not attached to the standard uncertainty.”

JCGM-WG1 will clarify.

65. KRISS 2 1.2 d)

The acronym “PDF” appears for the first time without its full expression.

To insert the full expression for PDF when it appears for the first time

Agreed.

66. PTB 6 1.2d) 2nd line

Abbreviation PDF is not explained at first use Substitute “deduce a PDF” by “deduce a probability density function (PDF)”.

Agreed.

67. LNE 17 1.2. d Page 2 line

No2

PDF is not specified here. The concept and the meaning are only introduced at page 4

… is used to deduce a probability density function (PDF) for the quantitative…

Agreed.

68. PTB 8 1.2 d) Last line

As this is a significant change it should be stated that the resulting Type A standard uncertainty is for a small number of indications often larger compared to the previous edition.

Add after the last sentence: “It also leads to larger standard uncertainties from Type A evaluations for small numbers of indication values.”

Agreed.

69. INTI 2 1.2 e)

The exclusion of the definitional uncertainty from the GUM scope. The exclusion of the definitional uncertainty reduces the GUM scope unnecessarily and may generate resistances to a more generalized application. We think that no requirements exist in the Bayesian theory to do that, when this component can be adequately included in the model. The GUM 2008 intended to be UNIVERSAL (GUM 2008, 0.4: “The ideal method for evaluating and expressing the uncertainty of the result of a measurement should be…universal: the method should be

To include the definitional or intrinsic uncertainty as a possible uncertainty component when i. it can be clearly stated in the

model and ii. the associated standard

uncertainty can be adequately known or evaluated

To exclude the definitional uncertainty from the GUM scope only when i. and ii. are not fulfilled



of 90

applicable to all kinds of measurements and to all types of input data used in measurements”. The aim of having an universal methodology seems to be partially abandoned in the draft. If cases when the measurand is not essentially unique are excluded, other document or supplement should be elaborated to contemplate items a – b of 8.3.3. According to the current GUM, the measurand in-self can be seen as an “intrinsic” random variable which mean is unknown in advance and which dispersion is characterized by the definitional uncertainty (GUM:2008 3.3.2.a - 3.3.2.b). This component must be known or evaluated in some manner, in order to include it in the LPU computation. Whereas, the new version explicitly discards last interpretation forcing to the measurand to be “essentially unique”. Ehrlich illustrates this (Figures 5 and 6 of Terminological aspects of the GUM, Metrologia 51, S145, 2014) saying that the meaning of the intrinsic pdf “becomes somewhat obscure”, because “there is no formalism for this type of analysis” within the GUM. Even accepting the rationale, we anticipate that this approach shall cause practical problems in many applications, in which components related to the definition of the measurand are traditionally accepted as a part of the measurement process and no confusion is caused by this perspective (the


of 90

definition of the measurand is the first stage of the measurement process. So, it is reasonable that the uncertainty caused by this stage was included) The exclusion of these components may cause problems to the GUM application in those fields As an industrial example, let us suppose that the diameter of a steel disk have to be measured and compared with specifications. Production failures may generate deviations from an ideal circle (form errors). If a procedure indicates n measurements, made over different directions, the definitional uncertainty can be statistically evaluated (in principle jointly with the measurement system repeatability, ANOVA tools could distinguish both components, if necessary). In this case, the definitional uncertainty (uncertainty due to the form errors) is not separated from components associated to the measurements in-self, But it can be included in the uncertainty analysis without any misunderstanding. In other cases, the uncertainty associated to the form errors can be evaluated only from the difference between the highest and the smallest measurement values. Of course, if the production process is under statistical control, an historical evaluation of the definitional uncertainty can be used instead of the Type A calculation All these strategies are valid but they assume that items a and b are included into the uncertainty analysis. We could force this example to be included in


of 90

the new GUM scope, defining the measurand not as the “diameter” (which is not essentially unique), but as the “mean diameter”. The mean diameter can be understood as the infinite average of measurements taken over all the possible directions Then, the above indications over n different directions can be interpreted as a statistical sample to estimate (or to evaluate) the mean diameter. However, this is a quite abstract and complicated way to include the form errors as an uncertainty component. For most of the industrial purposes, it has no sense to split components related to the production process (definitional uncertainty) from components related to the measurement process (“measurement” uncertainty, according to the draft) This differentiation may be very meticulous in theory, but practically problematic.

70. NMIJ 3 1.2 Clause e)

It is declared that definitional uncertainty is not considered in this edition of the Guide. However, the definitional uncertainty is sometimes a dominant uncertainty component in practice. There is concern that this might impair the practicality of this guide.

Add a procedure to include definitional uncertainty in uncertainty analyses.


71. CENAM 4 1.2, 3.4 many others

The “law of propagation of uncertainty” is used along the document, despite that it is an approximation, and not a “law” in any sense, a scientific one the least.

Please take this opportunity to leave behind calling the linear approximation a “law”.

The term is well-accepted, even though it is not a “law”.

72. CENAM 5 1.2

“There have also been major advances in computers and software for uncertainty calculation.” Cite at least one relevant reference to support

Cite reference to such major advances in software for uncertainty calculation.

Suitable references will be considered.


of 90

the statement regarding software for uncertainty calculation.

73. CENAM 6 2.1

The language in which this Guide is written is mathematically correct, even elegant, but unintelligible for the user generally found in a floor shop. Media to "translate" this document into a more friendly language for those users is in order. The time to develop those media will delay a while the implementation in the floor shops.

Consider means to make the GUM2 friendly for the floor shop user. As a partial solution, include more examples along the text as found in clause 7.1.


74. LNE 18 2.1

“this guide provides methods for evaluating and reporting MU that apply from the shop floor to fundamental research”. This guide is too complicated for shop floor, please change shop floor for production, it will be in accordance with the list.

“this guide provides methods for evaluating and reporting MU that apply from production to fundamental research”


75. PTB 9 2.2 Last but

second sentence

“This Guide does not provide advice on the development of a suitable measurement model: that task is specific to the metrological domain concerned.” Anyway, clause 8 is present – refer to this here.

Add afterwards another sentence “Anyway, clause 8 gives some information on measurement models.”

Clause 8 is not concerned with model development. That is the province of JCGM 103 under development.

76. CENAM 7 2.3

Some terms are used before definition is provided.

Include reference to definitions and terms in sections 5.X and 8.2.3 as needed.

Agreed. To be considered by JCGM-WG1.

77. NMIJ 4 2.5

Ordinal quantity is excluded from this draft. However, an example of uncertainty evaluation for Rockwell Hardness C scale (HRC) is provided in the current GUM. Also, international comparisons of HRC have been carried out, where their uncertainties have

Add a note describing that there are some ordinal qunatites, such as HRC, for which quantity-value scales have been established (see VIM3, 1.28), and that the GUM can be

Agreed. To be considered by JCGM-WG1.


of 90

been evaluated and utilized. It is not desirable to exclude all ordinal quantities.

applied to such quantities.

78. CENAM 8 2.6

“2.6 Knowledge about the measurand available before the measurement is not considered in this Guide.” This statement is not consistent with many basic and practical measurement techniques; e. g. a prior knowledge of the measurand is needed in order to choose the right measurement chain or to decide if it is either in steady-state or transient-state or if measurement conditions needed comply to proceed with its proper measurement or quantification. If the measurand is one of the input quantities the Guide allows the use of available knowledge, see clause 7.5.3 and clause 9.1.3. In indirect measurements, available knowledge about the input quantities in a measurement model means available knowledge about the measurand.

Delete otherwise clarify scope when measuring any of the input quantities.

Agreed. JCGM-WG1 to clarify text.

79. NMIJ 5 2.6,

9.2.6

In 9.2.6, the use of a pooled standard deviation is suggested. This is essentially a use of prior knowledge, and hence it is not consistent with the statement in 2.6.

Add a note on the relationship between the use of pooled standard deviations and the statement in 2.6 that no prior information is used in this guide.


80. NMIJ 6 2.6 Paragraph

No. 1/

Line No. 1

Obtained coverage intervals sometimes exceed the natural limits of quantities (such that a substance concentration cannot exceed 100 %), which causes troubles in interpretation of the results. Taking advantage of the Bayesian statistics, we can have a procedure to incorporate the natural limits of quantities as prior information in

Add a guide of estimating coverage intervals for quantities on which some natural constraints are imposed.

Agreed. A Bayesian guide is planned.


of 90

uncertainty analysis. 81. NMIJ 7

2.6 Paragraph No. 1/

Line No. 1

It is not clear whether prior knowledge should not be used even if it exists, or one can use it on his own judgment. We agree that use of prior knowledge should be avoided in this guide to make matters simple, but it is still desirable to clarify the policy on the use of prior knowledge to avoid future confusion.

Clarify the policy on the use of prior knowledge, and describe the reason behind that.


82. CENAM 9 2.7

The statement is not clear enough, and somehow surprising: It is generally thought that the MU is associated to a measurement result. However, the MU definition, adapted from the VIM, does not now refer necessarily to a measurand, but just to a quantity. Within this assumption, one interpretation of the 2.7 statement might be that the MU is associated to a measuring system, that according to VIM (3.2), deals with measuring instruments, leaving aside the procedures, environment and competences that are part of the influence quantities. Besides, it apparently associates MU to conceptual design and theoretical analysis, not to measurements.

Please revise the wording used in 2.7.

Similar to current GUM.

83. VNIIM 1 3. Summary

3.1 Paragraph 3.1 states, that the measurement result is reported in the form of an estimate of the measurand and an uncertainty or in the form of a coverage interval. This is inconsistent with the wording of paragraph 3.6.1 and paragraph 12.2.1.1, which state that the value and standard uncertainty should always be reported and, if necessary, coverage interval is to be given.

The wording of paragraph 3.1 should be brought into accordance with paragraphs 3.6.1 and 12.2.1. Figure 1 should also be corrected accordingly.



of 90

84. KRISS 3 3.1

3.2

Overall A measurement model could contain not only input quantities that denote uncertainty component, but also other parameters that do not denote an uncertainty component. It might be difficult for readers to discriminate between the two kinds by just looking at the measurement model only. It would be more advisable from the side of the users that the input quantities are identifiable by means of a glimpse of uncertainty equation before continuing to read evaluate individual input quantities. This is what actually is done in the current GUM.

Creation of a new subclause titled as “determining uncertainty equation” between 3.2 and 3.3

JCGM-WG1 is developing JCGM 103 to address modelling issues.

85. KRISS 4 3.1 Figure 1

In Step 2, input quantities are denoted by Xi (with upper case letter), whereas in Step 3, the measurand is denoted with a lower case character y.

It would be better to make two expressions consistent in format.


86. CENAM 10 3.1 Figure 1

Step 1

It is worth to note that the expression of the measurement model is a necessary but NOT sufficient condition for the measurement model can be linearly approximated. A more general, still valid model (VIM definition 2.48), that can be found in metrology, like that for implicit models such as humidity by measuring temperature, is h(Y, X1, X2, X3, … ) = 0, where h is a given function.

Include a note about this issue (implicit models) in 3.2.

Addressed in JCGM 102.

87. CENAM 11 3.1 Figure 1

Step 2

“Step 2: Evaluating the input quantities Xi, standard uncertainties u(xi) and covariances u(xi; xj)” Considering the relevant role that a PDF (or “state-of-knowledge" PDF) plays within a so called “Bayesian Approach”, it is suggested to make it explicit in Step 2 of the procedure for evaluating and reporting uncertainty.

Modify Step 2 to:

“Step 2: Evaluating the input quantities Xi, “state-of-knowledge" PDFs, standard uncertainties u(xi) and covariances u(xi; xj)”

PDFs are an important concept regarding both input and output quantities. They are omitted from


of 90

Subclause 3.3 leads the reader to Subclause 6.4 and Clause 9 where ‘Unless stated otherwise, PDFs in this Guide are “state-of-knowledge" PDFs.’ Furthermore, Subclause 8.3.6 clearly states that any user of this Guide shall keep in mind that “Knowledge of the input quantities is used to obtain PDFs for these quantities, from which can be obtained best estimates of these quantities and associated standard uncertainties, and,…”. In any event, users of this Guide shall address or deal with PDFs either known or unknown.

figure 1 for simplicity. They are a means to an end.

88. LNE 19 3.1 Figure 1

According to the previous comment, a last step should be added : “Evaluating the uncertainty budget”

Add an arrow going up from Step 5 to Step 1, called “Step 6 : Evaluating the uncertainty budget (3.7)”


89. LNE 20 3.1 Figure 1

The figure is very useful, But a) visually ‘step 5’ could be improved. b) The different clauses of the document

should refer to the different steps of fig 1 to help the reader structuring the evaluation

a) Stretch out the’ step 5’ from the stage of ‘step 3’ till the bottom of’ the ‘step 4’, to make correspond y and u ( y ) in front of step 3 and y + - Up [ylow, yhigh] in front of step 4 b) Refer to fig 1 throughout the document


90. CENAM 12 3.2

It would be convenient to provide an example where the measurement model as stated is an algorithm.

Consider the inclusion of an example.

A very good suggestion, but better in JCGM 110.


of Metrolo

gy”, Ukraine

2 3.2 -

The analysis of 3.2 shows that just as in the previous edition, there is no explanation in JCGM_100_201X_CD from where the mathematical ratio that determines the measurement equation - measurement model (1) in 3.2 is taken. It is not specified how to evaluate the accuracy of this mathematical relationship.

In 3.2 after the words “…known at the measurement function (5.16), whereas formula (1) is the measurement model 5.15” to insert the sentence: “Guidelines for the justification of the measurement model are given in Supplement 3 to the GUM (in progress).

It is clearly stated that the evaluation applies only to the measurement model under consideration. JCGM 103


of 90

We consider that this fact shall be commented in GUM, since the lack of information about the accuracy of the measurement equation does not allow the reliable evaluation of the methodological component of the measurement uncertainty (component of the measurement uncertainty due to inaccuracy of the measurement model) and casts some doubt on the correctness of the basic procedure of GUM - determination of uncertainty budget. If the corresponding explanation is supposed will be stated in Supplement 3 to GUM - Modelling (this Supplement is already in development for quite a long time), then it shall be, in our opinion, clearly stated in JCGM_100_201X_CD. Regarding to the acceleration of the preparation process of Supplement 3, we propose to consider the possibility to use in this document the method represented in the papers of NSC "Institute of Metrology", two of which were published in English: A.V. Prokopov. An Algorithm for Deriving the Measurement Equation and Estimating the Methodological Error In the Results for Indirect Measurements // Measurement Techniques, Vol. 48, No. 4, April, 2005, pp. 346-351. Available from: http://link.springer.com/article/10.1007%2Fs11018-005-0147-4 Aleksandr V. Prokopov. Justification for measurement equation: a fundamental issue in theoretical metrology // Cardiometry, No.3,

will address this point. JCGM-WG1 to check the given references.


of 90

November 2013, p.58-68. Available from: www.cardiometry.net/no3-november-2013/justification-for-measurement-equation A copy of one of these papers is attached.

92. CENAM 13 3.3

Any user of the new Guide shall bear in mind that quantification of best estimates for each input quantity implies that some information or assumptions shall be made regarding PDFs for each input quantity. Such information is stated and remarked in clause 12.3.

Modify sentence to: Accordingly, the standard uncertainty u(y) (5.17) is evaluated from the standard uncertainties u(xi) associated with the best estimates xi with associated “state-of-knowledge" PDFs and their covariance u(xi; xj)


93. VNIIM 2 3.4

In paragraph 3.4, the first line is presented inaccurately: «at the xi: …».

Paragraph 3.4 should be clarified as follows: «at the (x1 ,…, xn): …».

Accepted. JCGM-WG1 to clarify text.

94. LNE 21 3.4 Formula

(3)

You write standard deviation u(y) but the formulae is with the variance u²(y)

Write variance and explain that the law of propagation of uncertainty is in fact the law of propagation of variances and covariances

JCGM-WG1 considers this obvious and, in any case, the “law of propagation of uncertainty” is very well accepted.

95. LNE 22 3.4 Last §, line

1

Formula (2) has sense without conditions “Formulae (3) and (4) … “ Accepted. JCGM-WG1 to clarify text.

96. LNE 23 3.5 Page 4

“In general it should not be assumed that taking kp=1,96 gives 95% coverage”: how can we determine if PDF is Gaussian or not and how many results do we need to determine it?

Give more explanation and guidelines to help the user to decide

Clause 11 is already indicated, which gives some guidance.


of 90

97. CENAM 14 3.5.1

It is not clear the meaning of a “realistic coverage interval”. It suggests that there are others not realistic.

Clarify the use of the qualifier “realistic” despite of the explanation in 11.2.


98. PTB 10 3.5.1 Last

sentence “In general, this Guide is unable to provide realistic coverage intervals…”

Such self-defeating sentence is not helpful for the acceptance. It is also possible to calculate a realistic coverage interval with analytical methods, i.e. without using the Monte Carlo method (GUM-S1). The reader gets the impression that he/she does something wrong if he/she doesn’t use Monte Carlo.

Please change like “This guide provides realistic coverage intervals for two particular cases only, namely …”


99. METAS 3 3.5.1 Line 3

“In general, this Guide is unable to provide realistic coverage intervals, …”. So far, one of the major achievements of the GUM was to provide a realistic coverage interval. Therefore, it is not acceptable that the revised document puts this into question in such a strict way.

The statement should be weakened.


100. CEM 2 3.5.2 and 9.3.3

A reference to the case kp=2 for a 95,45% probability could be useful to be included

Already covered in 3.5.3,

101. CENAM 15 3.5.2

The literal Up has not been defined before using it.

Define Up . Accepted. JCGM-WG1 to clarify text.

102. CENAM 16 3.5.2

It seems that “Up”, formerly known as expanded uncertainty, does not bear now any denomination in particular. However, since the aim of a MU estimation is a value for the MU, or an interval for the measurand, it seems important to decide a name for “Up”.

Name “Up” somehow. Accepted. JCGM-WG1 to clarify text.

103. CENAM 17 3.5.2

To calculate the coverage interval, the new proposal recommends three ways a) knowing the PDF, b) when it is known that the PDF for

Include some guidelines to objectively decide if a PDF is symmetric and single-peaked or provide some reference.

Clause 11.2 already indicates where some


of 90

Y is symmetric about y, and is single-peaked c) When the PDF is not known.

guidance is given.

104. METAS 4 3.5.2

It will be generally understood, that for the majority of cases a coverage factor of k = 3 is to be applied. This will not be acceptable for industry.

The central limit theorem, which was used so far to justify a Gaussian distribution, should remain generally accepted to keep a coverage factor of k = 2 for a 95% confidence interval in the default case.


105. Inmetro 2 3.5.2

Facilitate reading and understanding. Include reference [39] in this item.

Clause 11.2 is already referenced and [39] is cited there. Clause 3 is a summary clause and the intention is not to clutter it.

106. LNE 24 3.6.2

Question: ‘two significant decimal digits’ is it equal to ‘two significant digits’? If we translate this sentence in French, it’s very different to add or not “decimal’

Write the good formulae in English

If binary digits were considered, the conclusions would generally be different.

107. INTI

3.6.2 and 12.2.1.5

Often, the model is intrinsically linear. In these cases, no approximation is necessary and no statement has to be made in certificates or reports

To replace the sentence “A statement should be made that any replacement of a non linear model by a linear has been judged to be acceptable” by the sentence “When a nonlinear model is approximated by a linear one, a statement should be made that the replacement has been judged to be acceptable”



of 90

108. LNE 25 4

Are JCGM 101 and JCGM 102 really “indispensable for the application of this document”? The reverse is correct, JCGM 100 is indeed indispensable for the application of 101 and 102

Remove JCGM 101 and JCGM 102 from normative references and perhaps add JCGM 104 instead

JCGM-WG1 to check normative references in final text

109. INTI

4.

Most of the users will apply the Guide without reading all the cited references. We want that its use will be as extended as possible (similar as for GUM:2008 or even more). Therefore, the GUM should not be conceived as a document for experts but for the metrological community in general, not only within NMIs, but also in the industry, commerce, etc The reading of all the references is not necessary

The following referenced documents are indispensable for the comprehension of the fundamentals application of this document.


110. CENAM 18 4,5 and 6

It is generally used to set up the normative references, terms and definitions, conventions and notation, prior to go into the development of the topic of interest. Although the clause 3 is intended as a summary, its understanding requires the background encompassing the normative references, like the VIM, the terms and the definitions, and the conventions and notation,.

Relocate clauses 4, 5 and 6 before actual clause 3.

That ordering was agreed by JCGM.

111. VNIIM 4 5. Terms and definition

The concept of «quantity» in this document is used in a slightly different, broader meaning than in VIM3. It would be useful to give its definition in Section 5.

The definition of «quantity», «influence quantity»/ «influence factor» and «correction» should be added. See also the comment to 8.3.4

A specific proposal is not made.

112. CENAM 19 5

The VIM is an essential tool for communicating in metrology. It may contain opportunities for improvement, however the

Avoid adaptions of VIM terms when are not really needed. Otherwise, include a note

Accepted. The VIM definitions will be used


of 90

GUM2 or any other document referring it, should not constitute a chance for updating or improving it. It is worth to note that VIM is being worldly used. So, modifications of terms from VIM just for the sake of language preference is not advised. As examples, the VIM definition for MU states that it is a non-negative parameter, which is now in a Note1. The “measurement error” definition adapted from the VIM, seems just to substitute the term “measured quantity value” by a newly defined “measured value”. No conceptual rationale is apparent. The value of these changes seems just by preference. When a modification of VIM terms is needed, an explanatory note would be very useful.

enlightening the change. or the rationale for any change given.

113. NMIJ 8 5,

6.4

Even though the concepts of "value of measurand", "random variable" and "uncertainty" are significantly changed from the current GUM, the differences are not explained in sufficient detail for readers not familiar with the Bayesian statistics to follow.

Add a section explaining in sufficient detail the differences in concepts and definitions that are caused by introducing Bayesian statistics.

Accepted. Bayesian aspects will be de-emphasized.

114. Inmetro 3 5

The guide, as outlined in subclause 2.4, can be used for both types of random variables: discrete and continuous.

Among the topics defined in the subclauses, we suggest the inclusion of two definitions: discrete and continuous random variables, since the guide, as outlined in subclause 2.4, can be used for both cases. After inclusion, complete the formulas ofsubclauses5.3 and 5.4 with the sums of the discrete cases.

To help keep the scope as simple as possible, discrete quantities are to be the consideration of a future JCGM document.


of 90

115. Inmetro 4 5

Correlation and statistical independence are concepts that tend to raise doubts among non-specialist readers.

Add a note explaining the difference between correlation and statistical independence, since these concepts are prone to raise doubts among non-specialist readers. This note would mention subclauses 6.5 and 7.4 of the guide.


116. NPL 1 5.1

Inclusion in the new GUM of a full definition of measurement uncertainty, early in the document, properly recognising the contributions for Type A and Type B uncertainties. (Without this it perpetuates the belief in some communities that measurement uncertainty is identical to the standard deviation of repeated measurements)

Adopt a more complete definition of measurement uncertainty, similar to that detailed in VIM 2.26.


117. NIM 1 5.1 and 7.1.1

“5.1 uncertainty parameter characterizing the dispersion of the values being attributed to a quantity, based on the information used” Meaning of “the values being attributed to a quantity” is not clear. “The values” here is neither identical to “the measured values” nor expressed in the same way as “the value of the measurand” in 7.1.1. 5.19 coverage interval interval containing the value of a measurand with a stated probability, based on the information available 5.20 coverage probability probability that the value of the measurand is contained within a specified coverage interval 7.1.1 The objective of a measurement is to

1) “The values being attributed to a quantity” should be further defined or clarified. 2) add a note to explain the difference between “quantity” used in the new version and the “measurand” used in the old version.

1) and 2) This is (close to) a VIM definition, which should be respected. The VIM definition refers to the measurand, which is regarded here (and in GUM-S1 and GUM-S2) as the output quantity. The definition has been adapted to “quantity” since we wished to apply it to


of 90

determine the value of the measurand (the quantity intended to be measured) (5.8). In general, the measurement produces only an approximation of the value of the measurand, which is called an estimate. NOTE The terms the value of the measurand and true value are viewed as synonymous in this Guide.” The “NOTE” in 7.1.1 is an important explanation for “the value of a measurand” in 5.19 and “the value of the measurand” in 5.20. So it is too late to appear in 7.1.1. Readers will not know whether it also applies to “the values” in 5.1.

3) To move this “Note” to a proper place in 5.

any quantity in the measurement model. When a definition is adapted we intend to give motivation for the adaptation3) Under consideration by JCGM-WG1.

118. LNE 26 5.3 First line

Clarification Add (PDF) Accepted. JCGM-WG1 to modify text.

119. INTI

5.4 Note 1

It is theoretically possible that E(|X|) = ∞ but E(X) is not well defined (for instance, Cauchy distribution)

A random variable might not have a finite expectation or it might not exist

Agreed. JCGM-WG1 will clarify the text.

120. LNE 27 5.5 Note 3

The notation u²(y) is used throughout the document to denote the variance. It would be useful to specify here such notation for the variance

Add a NOTE 3 : “The variance may also be denoted as u2(x)”

The comment is out of context. 5.5 is concerned with a property of the random variable.

121. Inmetro 5 5.8 to 5.22

Facilitate reading and understanding. Include figures illustrating the

definitions (see [8], for example) to facilitate the understanding of certain concepts and to avoid consulting the VIM [6] or other document for a better



of 90

understanding. 122. LNE 28

5.9 and document

To define the best estimate as the “expectation of the random variable describing the state of knowledge about a quantity” is an arbitrary choice that relies on the minimization of the mean squared error. Other choices of criterion could be made and would lead to different best estimates (for example median or mode). In Bayesian approach the MAP is commonly used as the location parameter

Justify the use of the expectation as the best estimate or allow the user to choose the appropriate parameter depending on the modelling and the purpose/goal of the uncertainty evaluation


123. VSL 12 5.9

Make users aware of difference with former practice

add Note: 'This means that the best estimate of the output variable is not only dependent on the best estimates of the input variables, but also on their uncertainties. It is therefore important to have realistic uncertainties for the input values in order to arrive at a realistic value of the output variable itself (and of course its uncertainty).'

These are meant to be terse definitions. However, JCGM-WG1 will consider using elsewhere in the document the suggestion.

124. VNIIM 3 5.9

Definition of the «best estimate» (through «expectation») is not entirely consistent with the definition of paragraph 5.4 and Bayesian approach. Perhaps it would be more correct to use the term «best value».

This needs clarification, perhaps in paragraph 7.5.1 regarding the use of the word «estimate» in the Bayesian approach.

“Best estimate” is used conditional on available knowledge. It is taken as the expectation.

125. CENAM 20 5.15

A computer algorithm may incorporate additional (or artificial) input quantities not necessarily coming from or attributable to the phenomena or measurand of interest. A computer algorithm may incorporate

Adopt definition given for measurement model in VIM. If necessary add a Note referring to computer algorithms.

“Computer” will be removed.


of 90

truncation or rounding error, besides numerical errors from filtering or signal conditioning operations of input quantities. Clause 7.2.6 warns about that and states “Measures of uncertainty are not intended to account for such mistakes.”

126. INTI

5.15 Measurem

ent model

“Mathematical relation or computer algorithm involving all quantities known to be involved in a measuremen” The VIM wording seems to be clearer than the definition here.

Measurement model: mathematical relation among all quantities known to be involved in a measurement or computing algorithm in which all those quantities are effectively involved”


127. VSL 13 5.20

Place note at most relevant place move Note 1 to 5.19 Accepted.

128. Inmetro 6 5.20

Notes 1 and 2 deal with the concept of coverage interval.

Remove Notes 1 and 2 of this subclause and insert them at the end of subclause 5.19, as both deal with the concept of coverage interval.

Note 1 to be removed.

129. CENAM 21 6.1

The statement “Occasionally, for strictly correct terms and expressions, typically long and elaborate, corresponding common-language versions, shorter and simpler, are also used.” Is not clear.

Use a clearer wording. No specific proposal.

130. Inmetro 7 6.2

Better nomenclature consistency between documents.

The letter g would be more suitable for the observation function, in accordance with articles published on the subject [10] and in view of a unique nomenclature for the supplements developed by JCGM.

GUM-S1 and GUM-S2 set a precedent.

131. Inmetro 8 6.3

Better nomenclature consistency. We suggest the adoption of the usual nomenclature in statistics, that is, upper case letters for random variables (Xi) and lower case letters for observed values

JCGM-WG1 gave much thought to the adopted notation.


of 90

of the same variable (xi1,xi2, xi3, …, xin) and not for the "best estimates" as explained in the guide. For general estimates the "hat" symbol is used on the estimator. In this case, 𝑥𝑥� would be more appropriate and would cause less confusion, that is, 𝑥𝑥�i = E [X]. This modification would be aligned with the example oft to temperature in Celsius.

132. NMIJ 9 6.4 and many other places

Both "uncertainty associated with ..." and "uncertainty about ..." are used throughout the draft, but it is hard for non-native speakers of English to distinguish the meanings between them.

Add some explanation on the difference between these two terms.


133. KRISS 5 6.4 Last

sentence

It is usual to use ux for expressing the uncertainty contribution rather than the standard uncertainty. Wouldn’t it bring confusion if sometimes ux is used for u(x)?

Use u(x) for expressing the standard uncertainty and ux for uncertainty contribution.

WG1 to consider.

134. VSL 14 6.5

Add explanatory formula r(xi,xj) = u(xi,xj)/(u(xi)u(xj)) Accepted.

135. CENAM 22 6.6

Incorporation or reference to conservative coverage intervals seems to be one of the major changes from GUM1 to GUM2. For the sake of a smoother change, a full annex may be compiled in GUM2 with sentences such as: Clause “3.5.2 Conservative coverage intervals …” Clause “6.6. … When considering conservative coverage intervals, kp and Up relate to a coverage probability of at least p.” Clause “11.2 Distribution-free coverage intervals”

Modify to: 6.6 The notations kp and Up denote a coverage factor and an expanded uncertainty, respectively, corresponding to some stipulated coverage probability p.

JCGM-WG1 wished to keep the document as short as possible .


of 90

136. Inmetro 9 6.7

Better nomenclature consistency. It is suggested the use of a clearer notation for the partial derivatives, ie𝜕𝜕𝜕𝜕/𝜕𝜕𝑋𝑋𝑖𝑖|𝑋𝑋𝑖𝑖=𝑥𝑥𝑖𝑖. Or, aligned with suggestion for subclause 6.3, 𝜕𝜕𝜕𝜕/𝜕𝜕𝑋𝑋𝑖𝑖|𝑋𝑋𝑖𝑖=𝑥𝑥�𝑖𝑖.

JCGM did not wish to make inessential changes from the current GUM.

137. LNE 29 6.10

The abbreviations listed are not of the same nature

abbreviations MAD and PDF should be set apart


138. LNE 30 6.10

The abbreviation MAD defined in 6.10 appears before its first occurrence in 9.2.7 where it is redefined

maybe all the abbreviations should be postponed at the end of the document


139. LNE 31 6.10 Page 9

The French abbreviations of the institutes or organisations should not been translated or in parenthesis

BIPM: Bureau International des Poids et Mesures (International Bureau of Weights and Measures) CGPM : Conférence Générale des Poids et Mesures (General Conference on Weights and and Measures) CIPM : Comité International des Poids et Mesures (International Comittee on Weights and and Measures)


140. Inmetro 10 7

Better reading and understanding. The inclusion of figures is suggested to facilitate understanding of the concepts covered on all subclauses.


141. CENAM 23 7.1.1

“7.1.1 The objective of a measurement is to determine the value of the measurand (the quantity intended to be measured) (5.8). In general, the measurement produces only an approximation of the value of the measurand, which is called an estimate.”

Modify to: “7.1.1 The objective of a measurement is to obtain a best estimate for the value of the measurand (the quantity intended to be measured) (5.8) and the associated standard

This is in the current GUM.


of 90

Determination of true values is a non-achievable task. A better framework is the one given in this same Guide in clause 7.6 … “…, the best estimate of the measurand and the associated standard uncertainty, and often a coverage interval for Y ….”

uncertainty, and often a coverage interval. In general, the measurement produces only an approximation of the value of the measurand, which is called an estimate with an associated standard uncertainty.”

142. KRISS 6 7.1.4

8.3.2

9.2.2

Influence quantity

The term “influence quantity” is used in these subclauses. I deduce the meanings of the term from the context of the subclauses. It seems to me that the implication of the term used in the revision is not exactly matched with that in VIM3.

Creation of an item in clause 5 describing the definition of influence quantity and some comments for the alignment of VIM3


143. VSL 15 7.2.2

What is a repeated indication? According to 5.12 systematic errors can also vary in a predictable manner and don't have to remain constant.

add explanation what is exactly meant by repeated indications


144. LNE 32 7.2.4

“uncertainty budgets” is not defined Either define the term or make a reference to the VIM


145. CENAM 24 7.2.5

Corrections for systematic effects should be encouraged, beyond the intention of “usually … required” is advised.

Modify the statement to: Corrections for systematic effects are required, …

It is too strong to make it a requirement.

146. CENAM 25 7.2.5

The practice of considering the educated guess merely as a contribution to the uncertainty about the measurand rather than to its estimate is deprecated (See also 7.5.2). When measuring systems are adjusted or calibrated using measurement standards or reference materials to reduce systematic effects, the uncertainties associated with such standards or materials are to be taken

Include a reference to example 6 in supplement 10.

Accepted. JCGM-WG1 to modify text.


of 90

into account to ensure measurement traceability [4, 9].

147. LNE 33 7.2.5 Line No. 6

“The practice of considering the educated guess merely … (see also 7.5.2)”: the meaning is not clear. Assuming that the authors want to underline that errors should be corrected when possible and not added linearly to the standard uncertainty, just say it in a simplest way

“Even if a correction is not fully estimated, the regular practice should be to apply the correction to the estimate of the measurand and to combine the uncertainty of the correction into the calculation of the uncertainty associated with the measurand. The practice consisting in adding linearly the estimated correction to the standard uncertainty of the estimate of the measurand (without forgetting the contribution of the uncertainty of the correction) should be avoided.” The information was clearly indicated in the previous version of GUM in the Note of the 6.3.1and F.2.4.5. Indications like the one in E.2.1 of the previous version of the document could also be of interest in the current version.


148. INTI

7.2.5

The sentence results confuse. The rest of the paragraph results sufficient to express the desired concept

To eliminate the sentence: “The practice of considering the educated guess merely as a contribution to the uncertainty about the measurand rather than to its estimate is deprecated”


149. NMIJ 10 7.2.5

Correction based on an educated guess lacks objectivity, and it has seldom been used in practice. Such a procedure might cause mistrust in the scientific validity of the GUM.

Delete the procedure using an educated guess in 7.2.5.


150. VSL 16 7.3.1

A pdf is more than that that gives -> that gives amongst others

Inappropriate proposal.


of 90

151. LNE 34 7.3.1 NOTE

The reference of the book [20] is not precise enough

Please, give the appropriate clause of the book. This comment could be generalized


152. Inmetro 11 7.3.2

Better reading and understanding. In this subclause it is suggested the reading of reference [36] for the elicitation of joint probability distributions for inputs that have some degree of dependence. In clause 9,Table 2,one can find the most common cases, where there is guidance for elicitation from the type of existing information. Why do not complete Table 2 with instructions on how to elicit this joint distribution? Although it is not a general case, a common habit among researchers who encounter this problem is to consider that the input variables are independent of each other, which does not necessarily reflect the reality of the problem. Some guidance on how to explicitly elicit the joint distribution (and not as a reference only), would facilitate understanding and would encourage the use of this approach.


153. VNIIM 5 7.4 Expectation, standard deviation and covariance

7.4.1 7.4.1 conflicts with the definitions from 5.4 (5.5). Mathematical expectation and variance are not always the parameters of the PDF in an explicit form.

The inaccuracy of the wording in 7.4.1 should be corrected.

No specific proposal made.


of 90

154. LNE 35 7.4.1

Expectation and standard deviation may not be natural parameters of the pdf except in special cases (Gaussian pdf for instance); for instance the parameters of a bounded uniform pdf are the lower and the upper bounds

Say instead: the expectation is a measure of location…, the standard deviation is a measure of dispersion

The current wording says that already.

155. LNE 36 7.4.2

Covariances are used for two quantities which are linearly dependent. But other types of dependence exist between two quantities (for example, non-linear relationship)

Explain why this guide considers only the covariances and make this assumption clear

No specific proposal made.

156. CENAM 26 7.5

The paragraph looks somehow difficult to read, but for those familiar with mathematical statistics.

For sake of friendliness, highlight the corresponding guidance by writing it in a concise way in first place, and then write its explanation.


157. Inmetro 12 7.5

Better reading and understanding. The purpose of the first two subclauses (7.5.1 and 7.5.2) is to explain the use of expected value as the best estimate, since this is the amount that minimizes the mean square error MSE = E[(X-E[X]²]. We suggest clarifying the overall goal of subclause 7.5, as it appears to be a gathering of concepts without cohesion, especially the first two subclauses of 7.5. Another suggestion is to make clear, from a statistical point of view, the amount to be minimized (MSE). It would be possible to previously define this concept (on subclause5) so that the reader would previously know the context of the problem.


158. Inmetro 13 7.5.1

Better reading and understanding with nomenclature.

In order to facilitate understanding and avoid doubt

Accepted. JCGM-WG1


of 90

it is suggested to rewrite the first part of equation (5) as follows: 𝐸𝐸[(𝑋𝑋 − 𝜉𝜉)2] = ⋯, with the brackets. In addition,we suggest the usual x, instead of the symbol𝜉𝜉 as the result of a measurement (data).

to modify text.

159. MSL 2 7.5.1 P1/L6

Unnecessary word Remove “possible” Accepted. JCGM-WG1 to modify text.

160. PTB 11 7.5.2

In practice sometimes uncorrected estimates have to be used. See also http://www.ptb.de/cms/en/fachabteilungen/abt8/fb-84/ag-840/publika-840/jcgm-1040.html and Lira I / Wöger W, Meas. Sci. Technol. 9 (1998) 1010-1011

Substitute “deprecated” by “should be avoided” to allow the use in exceptional cases.


161. INTI

7.5.2

The following paragraph would need to be illustrated: “If for some reason an estimate of X is chosen to be ξ = x + b, where b is a known non-zero shift, then from expression (5) the squared measurement uncertainty associated with ξ will be u2(x) + b2 > u2(x). In addition, the pair ξ = x + b and the associated uncertainty cannot be used for purposes of uncertainty propagation. For these reasons, the use of an estimate other than ξ= x is deprecated”

To include an example in which both options are applied, the “deprecated” one (x + b), and the “proposed” one (x). For instance, in the use of an instrument in which a previous calibration allowed to know a systematic effect b

The concept is so simple that its illustration would serve little purpose.

162. NMIJ 11 7.5.2

In industry where economic efficiency and simplicity are important, a measured value whose known bias is not corrected is sometimes more useful than a value with the bias corrected. Also in statistics, a biased estimate is not necessarily a wrong estimate.

Add a guide for evaluating the uncertainty of a measured value of which a known bias is not corrected.


http://www.ptb.de/cms/en/fachabteilungen/abt8/fb-84/ag-840/publika-840/jcgm-1040.html

http://www.ptb.de/cms/en/fachabteilungen/abt8/fb-84/ag-840/publika-840/jcgm-1040.html


of 90

It will be valuable to show how the uncertainty of such a biased estimate can be evaluated.

163. PTB 12 7.5.3

“… sometimes called a Type A …” and “… sometimes called a Type B …”. These were the terms in the previous edition – this should be pointed out.

Substitute “sometimes” by “previously”.

JCGM-WG1 to modify text.

164. LNE 37 7.5.3

The expressions “sometimes called a Type A evaluation of uncertainty” and “sometimes called a Type B evaluation of uncertainty” are to be changed. They are not “sometimes “called like that. It is the GUM itself in its current version which is at the origin of these names (type A and type B) that we are supposed to use when applying the GUM. The new proposal must clearly state if these notations (type A, type B) are maintained or not, if they are compulsory or not, and if other names are proposed. Anyway this classification of uncertainties, whatever the names, should be maintained. Practically, this allows to identify and evaluate the uncertainty components whose value can be reduced by making repeated measurements and those on which this repetition is ineffective. This allows planning experiments by optimizing the number of measurements to perform (neither too much nor too little) in terms of uncertainty, especially in cases where one or more type-B uncertainty components are significant or even predominant

General revision of the text after the study of the possibility of minimizing a priori assumptions, when it is possible, i.e. by using robust estimators of the variance of input quantities obtained after repetition of measurements Keep the classification type A and type B evaluations of uncertainty

JCGM-WG1 to modify text.

165. CENAM 27 7.5.4

The statement seems to be dealing with a general issue.

Relocate the paragraph to clause 6 and provide the reference to the supplement where the treatment will be available.



of 90

166. NMIJ 12 7.6

It is claimed in 7.6 that the measurement result is given in summary form as the best estimate of the measurand and the associated standard uncertainty, and the best estimate is defined in 5.9 as the expectation of the state of knowledge PDF. This implies that the expectation of the PDF should be chosen as the measured value. However, the choice of the final measured value should be left to metrologists. A choice which is scientifically most reasonable is not necessarily the best one when economic efficiency or procedural simplicity is important.

Replace "the best estimate of the measurand" with "an estimate of the measurand" in 7.6. Avoid setting a rule to determine a measured value only from statistical viewpoint.


167. Inmetro 14 8.2

Better reading and understanding. It is very informative to give examples of measurands, but it would be even better to use the examples to show the difference between linear and non-linear functions. Also the examples could highlight the number of input variables in each of them and, if possible, insert a multivariate model example in contrast to the univariate model(although it is not the scope of this guide but of the GUM-S2 [2]).

The proposal would extend the document too much.

168. LNE 38 8.2.1 line 3

missing word “the” …is a prerequisite to the estimation…


169. CENAM 28 8.2.1

There are also univariate models, as defined in 8.1.1, although not necessarily linear ones, that takes forms of expression different from ec. 1, namely as h (Y, X1, X2, X3, … ) = 0 , where h is a known function.

Revise the statement. Covered in JCGM 102.


of 90

170. MSL 3 8.3.1 P1

A short example (perhaps a footnote) would help explain what is meant by “… even if their contribution to the estimation of the measurand is negligible”


171. VSL 17 8.3.1

uniformity of vocabulary, term 'meaningful' unclear in this context

'meaningful components' -> 'non-negligible contributions'


172. PTB 13 8.3.1 First

sentence

“… that can contribute meaningful components of uncertainty …”. It is not clear what “meaningful” means – please explain.

Substitute the half sentence „even if their contribution to the estimation of the measurand is negligible“ by „Input quantities with negligible contributions shall be disclosed in 12.3.” Add in 12.3 c) “including quantities regarded as negligible”.


173. MSL 4 8.3.2 P1/L2

It is the indication value(s) that are intended here

Re-word to “… a single indication value, repeated indication values, or …”

The wording is correct.

174. INTI 8.3.2

Other combinations are possible. For instance: in order to determine the instant air density in a laboratory, the temperature, barometric pressure and relative humidity have to be known. The values of these influence quantities are determined through measurements. But the associated uncertainties are previously assigned, on the basis of external sources, as resolutions, accuracy classes, calibration certificates, etc, of the instruments used for carrying out those measurements

8.3.2 The input quantities may be categorized as - quantities whose values and uncertainties are directly determined in the current measurement. These values and uncertainties may be obtained from, for example, a single indication, repeated indications, or judgment based on experience, and may involve the determination of corrections to instrument readings and corrections for influence quantities such as ambient temperature, barometric pressure and humidity; and - quantities whose values and uncertainties are brought into the measurement from external

Current wording is adequate to cover the “other combinations”.


of 90

sources such as calibrated measurement standards, certified reference materials and reference data given in handbooks, and -a combination of both possibilities

175. INTI

8.3.3

See comment to 2.1.2.e. To include 8.3.3. a – b into the GUM scope, in cases when the components associated to the definitional uncertainty can be clearly included in the model and quantified



of Metrolo

gy”, Ukraine

3 8.3 8.3.3

see NSC comment N°2

The inaccuracy of measurement model (1) in 3.2 is not specified as a source of uncertainty in the list of sources of uncertainties.

Add to 8.3.3; n) inaccuracy of the measurement model


177. VNIIM 6 8. Modelling the measurement

8.3.4 The tem «input quantities» in the second line is used incorrectly.

The term «input quantities» should be replaced with «influence factors».

The term is used in the current GUM, GUM-S1 and GUM-S2.

178. CENAM 29 8.3.4

There might be some attributes, rather than variables, that may impact the MU value but whose quantification is senseless or very difficult, like the cleanliness of a weight or the air dustiness. Their effects on the MU are minimized by applying appropriate procedures.

Add a paragraph at the end, as guidance like: There are some attributes that may impact the MU value but whose quantification is senseless or very difficult, like the cleanliness of a weight or the air dustiness. Their effects on the MU are minimized by applying appropriate procedures.


179. LNE 39 8.3.4

The use of cause-and-effect diagram does not illustrate only the fact of grouping several input quantities, it’s always very useful for

Introduce cause-and-effect diagram in clause 8.1.1 as a useful tool to write the

Under consideration by JCGM-


of 90

evaluating measurement uncertainty measurement model to provide an exhaustive list of uncertainty sources and check for possible redundancy

WG1.

180. INTI

8.3.4

The sentence “In such cases, the effect on the measurand of several input quantities can be evaluated as a group” could be better illustrated

To include an example where several components generating random effects are merged in a unique component. For instance, if the length of a rectangular piece has to be measured n times with a caliper, the geometrical irregularities of the piece, the random effects of the instrument, effects associated to alignment variations, and random effects associated to the readings, can be commonly associated to the repeatibility of the measurement system and statistically evaluated as a whole


181. PTB 14 8.4 Starting

with “An important example”

The scope is much too narrow. It is more consistent to propagate Bayesian approaches for these exceptions – while LSQ and ISO are non-Bayesian.

Delete clause 8.4 starting “An important example”.

Accepted.

182. INTI 8.4.

a) From our point of view, the concept of observation equation is a relevant contributions in this version. It provides a necessary bridge between metrological models (or “models associated to the measurement”, as defined in the current version) and statistical ones. Then, a considerable statistical background is incorporated in the GUM scope. Particular, linear and non linear regression models are extremely useful. b) In this context, we interpret that the “measurement model” concept is aligned

a) An example relating least square calculations could be incorporated in the Guide, anticipating a future JCGM Supplement (for instance, arising calibration of weights by the subdivision method) b) We suggest that the GUM

a) For a future GUM Supplement. b) JCGM-WG1 wanted a simple treatment. Numerical methods “convert” h(…) = 0 into Y = f(…): an


of 90

with the VIM definition (2.49) “measurement function”. It covers the cases when the measurand y can be derived from the input quantities through an explicit equation y = f(x1, …, xn). On the other hand, the “observation equation” is a special case of the more general “measurement model” (VIM 2.48). Here, an implicit relationship between the measurand and the input quantities is only available h(y, x1, …xn) = 0.

refers to the correspondence between both type of models and the VIM definition

algorithm.

183. Inmetro 15 8.4

Better support with the inclusion of references.

It would be interesting to refer to Supplement JCGM 107 (Applications of the least-squares method) while giving examples of the evaluation of uncertainty in calibrations. Still on the subclause 8.4, it is suggested to add another reference than [37] on modeling via observation equations.

See response to 182.

184. NMIJ 13 8.4

9.2

8.4 Paragraph No. 1/

Line No. 1

Evaluation of homogeneity and stability of materials is imperative in reference material development, in which ANOVA methods play an essential role. ANOVA is referred to in 8.4 of this draft, where the ISO 5725 is referenced. However, the ISO 5725 is not based on the Bayesian statistics. A guide to ANOVA methods based on the Bayesian approach is needed.

Add a guide to ANOVA methods based on the Bayesian approach in 9.2.

An ANOVA example will be included in JCGM 110.

185. LNE 44 9.1.2 Table 2

The graph of ‘scaled and shifted t’ is thinner than the Gaussian one. It’s very important to have an illustration in accordance with calculation

Widen the graph of Student t more than the Gaussian


186. LNE 40 9.1.2 Note

Redundant with 7.3.2 “various method’s including Bayes’s rule…” that already points to table 2; moreover, introduces the prior

The note should be deleted or augmented, or at least precise which pdfs in table 2 result exclusively from Bayes’s rule

Accepted.


of 90

terminology without defining it Define the prior terminology 187. CENAM 30

9.1.3 The statement “Users of this Guide should be aware that not considering covariance terms produces the same effect as assuming they are zero, which requires justification” seems unnecessarily wordy.

Modify to: Users of this Guide should be aware that not considering covariance terms, or assuming them zero, requires justification.

Accepted.

188. CENAM 31 9.1.4

There are situations where a duplication of a contribution to MU is needed, although such contribution appear at different stages: When an indicating measuring instrument is calibrated, the instrument resolution contributes to the MU, which is then stated in the calibration certificate. When the same instrument is used to measure, the MU of this result includes the instrument resolution once more.

Include a statement looking forward necessary double-counting of MU contributions.


189. VSL 21 9.2

Many applications in metrology rely on the use of regression analysis. Nothing is said about regression analysis, and how it can be used in the framework of the GUM. Clause 9.2.3 gives a very specific treatment of calculating a standard uncertainty from a series of repeated observations. It is reasonable to expect that such a specific treatment also applies for fitting models through data. Guidance is lacking.

Provide guidance on how to use results from regression in the setting of the GUM.

A JCGM least-squares document is being developed..

190. VSL 22 9.2

Analysis of variance, treated in JCGM 100:2008, is missing

Add guidance on analysis of variance and how it should be used in the framework of the GUM

An ANOVA example will be included in JCGM 110.

191. LNE 41 9.2

A type A evaluation can be performed for: - evaluating the mean value and the uncertainty associated to the mean value, because the best estimate of xi is the mean value (1)

Explain this point because this is really difficult for users to make difference between “s” and “s/√n. and put a simple example of the two possible cases



of 90

- evaluating the dispersion of a parameter, because the best estimate of xi is an unique value (2)

192. LNE 42 9.2

The multivariate t distribution is used in JCGM-110:201X-CD §5.3.2 but is not presented in the proposal, in particular in sub clause 9.2

Add a subsection dedicated to multivariate analysis of measurements in 9.2


193. INTI 9.2

The n=2 / n=3 cases . The Bayesian approach, as applied in the draft, requires a higher number of observations than in the previous version. It is illustrated in the example 5.3.2. in JCGM 110, which requires n = 6, while the same example in GUM:20008 required n = 5. It can be a serious problem in routine measurements in which few replicates are employed, and no historical evaluations of repeatability can be made. Many cases of that can be found in Chemistry or Microbiology. In them, the lack of homogeneity of test samples is a significant source of uncertainty. Strictly speaking, they are also cases where the measurand is not essentially unique. But here, additional problems are presented. Often, n = 2 or n = 3 replicates are considered enough for routine tests. Then, Type A evaluations with uninformative Jeffreys prior cannot be used to assign the associated uncertainty. Considering cost/benefit reasons, it has no sense to increase n: the target uncertainty is relatively large. Historical pooled estimations are not correct either because the random effects are

A practical option should be developed in which the requirement n > 3 may be relaxed in routine measurements. For instance, by using an informative prior to the homogeneity component instead of the Jeffreys one.



of 90

dispersion is related to the test sample rather than to the measurement system. The draft does not give solutions to these cases. In the last years, a considerable effort has been made in order to include chemical, biochemical and microbiological laboratories into the metrological community. The use of the GUM (or of other documents derived from it) must be a relevant part of that inclusion. The new GUM cannot be an obstacle.

194. NMIJ 14 9.2

To use a pooled standard deviation, if available, is surely a preferable solution to the case in which the number of repetition is less than 3. However, it is not always available. If no means is provided to handle n ≤ 3 cases, there would be a serious confusion in situations where n ≤ 3 cases have been successfully handled in the framework of the current GUM.

Provide a guide to handle n ≤ 3 cases other than using a pooled standard deviation.


195. Inmetro 16 9.2.1 Table 2

Better reading and understanding with a change in notation.

Improve assessment of the exponential function, in the second column of the last row of Table 2, from Ex to exp.


196. CENAM 32 9.2.1 Note

These features are important enough to be highly visible, and expected to be part of the Scope.

Relocate the Note to the Scope. Accepted.

197. PTB 15 9.2.1 Note

It is more consistent to propagate Bayesian approaches for these exceptions – while LSQ is non-Bayesian.

Substitute “least-squares” by “regression”.

A JCGM least-squares document is being developed which may include classical LS and Bayesian


of 90

approaches. 198. INTI

9.2.1 Note Applications of least squares analysis are mentioned in 8.4.and it has no sense to exclude them, even though that other documents can give a more detailed treatment in the future. Mass calibrations by the subdivision method, chemical measurements made with equipments calibrated through calibration lines or curves (chromatographers, spectrophometers, etc), among others, are very useful cases that should not be unnecessarily excluded.

NOTE Instances beyond the scope of this Guide include (i) a sequence of repeated indication values, for which their order is relevant, implying correlation between the values, and (ii) calibration data describing the dependence of one quantity on another. Such cases are treated by time-series analysis, spectral analysisis and Allan variance, and least-squares analysis, respectively

Accepted.

199. NMIA 2 9.2.1 Note

The previous GUM gave an example of time-correlated data and a reference to Allan Deviation so the new version is less helpful.

Give frequency measurements as an example of time-correlated data


200. NMIJ 15 9.2.2

Paragraph No. 1/

Line No. 1

This draft guide restricts Type A evaluation to repeatability conditions. However, the effects of measurement days and operators are often important.

Allow Type A evaluation based on data obtained under conditions other than repeatability conditions. Add a guide on how to do this.

A statement is included in the scope.

201. Inmetro 17 9.2.2 and 9.2.3

Better reading and understanding with a change in notation.

Again it is suggested a change in notation. Instead of 𝜉𝜉𝑖𝑖,𝑟𝑟, use 𝑥𝑥𝑖𝑖,𝑟𝑟 as indicated values. Also changing the best estimates by the use of the “hat” symbol.


202. CENAM 33 9.2.3 and 9.3.4

The use of shifted and scaled t-distribution to build the coverage factors is not clear

Include the adequate coverage factors for shifted and scaled t-distributions, similar to section 9.3.3 for normal distributions.

This clause is not concerned with providing coverage factors. Clause 9.3.3 uses coverage factors to obtain standard


of 90

deviations. 203. LNE 45

9.2.3

& 9.2.8

Eq. 11

Eq. 15

In a Bayesian setup, these choices result from the marginalization of the joint posterior distribution of the mean and standard deviation/variance using the non-informative Jeffrey’s prior. The results depend on the choices of prior for the mean and standard deviation. Other factors than (n-1)/(n-3) could be obtained depending on a different choice of prior. The assumption made here should be clearly stated and justified and guidance should be given to the user to compute posterior distribution in different situations

For consistency, the choice of prior should be mentioned and justified. A note should be added to give guidance to the user in order to compute a posterior distribution in general


204. NMIJ 16 9.2.3

The multiplier √(n-1)/(n-3) is applicable only when the measurement data follows a normal distribution. A guide is needed on how uncertainty can be evaluated when the distribution is considered not normal.

Add a guide on how "Type A" evaluation is conducted when the assumption of normality is inappropriate.

JCGM-WG1 to consider clarifying the text.

205. NMIJ 17 9.2.3

There is no description on how prior distributions are chosen. It is desirable to describe that specific priors are used to express the fact that we have no prior information, so that this draft can be better understood by people not familiar with Bayesian statistics.

Add a note in 9.2.3 that the t-distribution is derived from a specific (i.e. Jeffreys) type of non-informative prior distribution.

JCGM-WG1 to consider clarifying the text.

206. NMIJ 18 9.2.3 and 9.2.9

The formula (11) for a univariate Gaussian distribution and the formula (16) for a bivariate Gaussian distribution provide different standard uncertainties. The reason for this needs to be explained.

Add an explanation on the reason why the standard uncertainty of a quantity changes merely by simultaneous measurement of another quantity.


207. CENAM 34 9.2.5

The standard uncertainty evaluation for n=2 and n=3 when there is no previous experience, is important in many applications like destructive tests, and other non-

It could be possible to include in the GUM a way to evaluate the standard uncertainty for these cases based in the article: “KACKER, R., AND JONES, A.

JCGM-WG1 to consider the case n < 4


of 90

repeatable phenomena. On use of Bayesian statistics to make the Guide to Expression of Uncertainty in Measurement consistent. Metrologia 40 (2003), 235-248”.

208. VSL 18 9.2.6

It is doubtful if it is typically a 'pooled' std dev. In many cases just one large batch will be evaluated and s_p determined from it (which is thus not pooled).

Reformulate. 'typically' -> 'sometimes' or 'may be'

Accepted.

209. VSL 19 9.2.6

For some measurement the standard deviation may be very dependent on the artefact. For economical, technical or ethical (medicine) reasons still only 3 measurements are taken. What to do in this case? Possible example (unverified): determination of straightness deviation. Examples will certainly pop-up in practice after some time.

Give additional guidance for the case n <= 3 and s_p not available, a kind of emergency solution.

Accepted

210. VNIIM 7 9. Evaluating input quantities, …

9.2.6 In paragraph 9.2.6, formula (12) is given as the main one, although in practice formula (13) seems to be used as the main.

In paragraph 9.2.6, it is advisable to indicate formula (13) as the main one.

Accepted. Make the note containing (17) a subclause at the same level as 9.2.6.

211. CENAM 35 9.2.6

Often, there appear situations where it is not possible to measure under strict repeatability conditions, like in calibrating flow measuring instruments.

Include some guidance addressing the issue.


212. PTB 16 9.2.6 Line 7

In a number of measurement processes the characteristics of the object to be calibrated strongly influence the uncertainty, e.g. form deviations on dimensional calibrations. The characteristics of different objects, therefore, should be the same or they should affect the uncertainty in the same manner.

Substitute “measuring system” by “measuring system under the same conditions or the measurement objects which have the same characteristics affecting the uncertainty in the same manner”



of 90

213. PTB 17 9.2.6 Eq.

(11)+(12)+Note 2

There is non-consequent use of “n-1” and the term degree of freedom ν for the pooled standard deviation. Although it is inherent in the text body, the reader has to conclude for himself that in all other equations the Guide is talking about degree of freedom ν = n-1. This is also the case in connection with table 2

Assure stringent usage for term degree of freedom in the notation and equations.

Accepted. Text to be modified.

214. PTB 18 9.2.6 Sentence

starting with “A practice ...” after Eq. (12)

It is not clear for what purpose the comparison described shall be used.

Add “Similarity of these two standard deviations would assure appropriateness.”

Accepted.

215. NMIJ 19 9.2.6

The equation that follows Equation (14) is not assigned a number, even though it is cited in the example document (JCGM110 :201X CD).

Assign a number to this equation.

JCGM 110 refers to current GUM.

216. NIM 2 9.2.6 EXAMPLE

The repeatability standard deviation and repeatability limit stated in standard methods is not suitable to be directly used for standard uncertainty evaluation. It’s just used as a request for data quality in many cases while method verification or validation is necessary before using them for uncertainty evaluation. The description this example should be cautious to avoid misunderstanding. We suggest adding a sentence to limit the use of these data stated in standard methods.

Two test values for the mass fraction of sulphur obtained successively by the same operator and instrument on aliquots prepared from the same test sample of diesel fuel were 17.1 mg kg-1 and 17.7 mg kg-1. The repeatability standard deviation reported in the method (ASTM D7039 [2]) is 0.32 mg kg-1, and the repeatability limit is therefore = 0.91 mg kg-1. After method verification or validation, these data can be used for uncertainty evaluation. Because the difference of 0.6mg kg-1 falls within the repeatability limit, the measurement result is (17.1 + 17.7)/2 = 17.4 mg kg-1 with standard uncertainty

mg kg-1 (or 0.2 mg kg-1 to the same number of

Accepted.


of 90

decimal places as the estimate) 217. LNE 46

9.2.6 Example, last sentence

“(or 0.2mg.kg-1 to the same number of decimal places as the estimate)”. One rounds the estimate in function of the uncertainty and not the uncertainty in function of decimal places of estimate

Remove the sentence under brackets

Accepted.

218. VNIIM 8 9.2.7

In paragraph 9.2.7, it is recommended to use robust estimates: median for location, MAD for standard deviation. However, after that it is proposed to pass on to the option 1 from the table 2, i.e. to further expand the standard deviation.

In paragraph 9.2.6, using robust estimates it is advisable to go by the option 4 from the table 2, i.e. not to expand the standard deviation further.


219. Inmetro 18 9.2.7

Provide guidance on identification of outliers and the use of the median.

We suggest the inclusion of a reference for identification of outliers in a data set, providing guidance to readers not familiar with this type of situation. A good example is the ISO 16269-4: 2010 [11]. We also suggest improving the example, with more concise comments, justifying the use of the median based on the experience of the analyst.

Accepted.

220. NIM 3 9.2.7 EXAMPLE

Robust statistics can be used for uncertainty evaluation. But this example does not support the text in 9.2.7 very well: 1) By using Dixon and Grubbs rules, there

seems no outlier among the indication values. Also the standard uncertainty of 1775 is smaller than standard uncertainty of 2189 evaluated by the way of standard deviation according to 9.2.2 and 9.2.3. In this case, the rationality of uncertainty evaluation by robust statistics will be questioned.

2) The number of estimates in this example

Change the example so as to support the text in 9.2.7 more reasonable.

Accepted.


of 90

is too limited for reliable evaluation of uncertainty. Actually in PT schemes, the number of participants is usually more than 15 for application of median and MAD.

221. PTB 19 9.2.8

Two parameterisations of the Gamma distribution exist. It is not clear which is used here.

Better define Gamma distribution or give reference (on p. 38).


222. CMI 2 9.2.8

The concept and the example are not very clear, specially the concept itself, being not very user-friendly. It should be clear to any reader, being a physicist or chemist, without using the references.

To make the whole paragraph more self-explanatory or to put it out.

No specific proposal.

223. CMI 3 9.2.9

Basically, only formulas are given without any derivation.

To add the derivation of the formulas.


224. MSL 5 9.2.9 P1

This example needs further explanation. It appears that u(x_i) and u(x_j) are standard uncertainties associated with components of a joint PDF for the bivariate RV and that this PDF is of the form described in GUM-S2. Reference [26] cited here does not help in understanding equations (16). From [26] one might reason that a term (n-3) should appear in the denominators of (16), because [26] identifies the multivariate-t distribution with least uncertainty as the so-called Geisser-Cornfield case, which has Student-t marginals with dof=n-1. So perhaps GUM-S2 should be cited here in place of [26]. It seems strange, however, to cite S2, which deals with multivariate measurands, whereas this document is intended for univariate measurands only. For this reason, we

Reconsider this section, its references and its relevance to this document.



of 90

expected to find a more appropriate citation from GUM-S1 but were unable to do so.

225. MSL 6 9.2.9 P1

A note could be added that there is also an example in JCGM-110 related to this example.

The plan is that JCGM 110 cites JCGM 100, and not the other way round.

226. VSL 20 9.2.9

Very strange result: if you measure a length and a width, you get higher uncertainties than when individually evaluating the length and the width

Add a note/comment justifying result or reconsider 9.2.9 completely


227. CENAM 36 9.2.9

Equations 16 and 17 use (n-4) in the denominator This is misleading. NOTE 2 wording suggests that the degrees of freedom depend on the number of simultaneous measurements and the number of measured quantities. References: Theorems 3.3.2 and 7.2.1 at “Multivariate Analysis” by Anderson, 1948 Theorem 6.1.1 at “A First Course in Linear Model Theory” by Ravishanker & Dey, 2002.

Use (n-3) for the denominator in equations 16 and 17.

These equations apply for n>3.


228. VNIIM 9 9.3

Paragraph 9.3 provides a number of practicable options, for which methods of assigning means (best estimates) and associated uncertainties are specified. However, paragraph 9.3.8.3 further provides modelling using the t-distribution with an undefined number of degrees of freedom, which is illogical.

It is advised to either delete the text of paragraph 9.3.8.3 (preferably), or provide a detailed explanation (justification) and include this option in table 3 with this justification.

The contexts are different.

229. VSL 29 9.3 table 2 and

difference between 'elicitation' and 'available knowledge' not very clear

merge table 2 and 3 and insert 9.3.8 in 9.3

A JCGM-WG1


of 90

3 decision to have two tables.

230. LNE 47 9.3 Table 3

Sometimes intervals endpoints are available, with probability greatest at one of the endpoints. This results in an asymmetric triangular distribution which is not addressed here.

Suggest to replace the symmetric triangular distribution with the triangular distribution using a third parameter c (a ≤ c ≤ b), which is more general and includes the symmetric distribution when c is the midpoint.


231. PTB 20 9.3.1

The transition from details in 9.2 to the sentence “Relevant information ...” in 9.3.1 is quite abrupt.

Substitute by an introductory sentence like “Knowledge about the uncertainties of input quantities may also originate from other sources than from the statistical analysis of data. Relevant information ...” Before “Table 2 facilitates...” add “This guide gives a list of the most common forms of knowledge and the corresponding PDFs.”

Accepted.

232. PTB 21 9.3.3 Third line

Term standard deviation is mixed with symbol u(x) [standard uncertainty], a confusion of the principal symbols may occur.

Delete the half sentences “with expectation x and standard deviation u(x)” and “with expectation x and standard deviation Up=kp” or adopt Supp 1 wording (6.4.7).

Accepted


of Metrolo

gy”, Ukraine

4 9.3 9.3.3.

In Clause 9.3.3 lists the coverage factor corresponding to different probabilities, but CIPM MRA prescribes to estimate the expanded uncertainty of only for probability 0.95.

To estimate the expanded uncertainty of only for probability 0.95.

That is the prescription of the MRA, but other prescriptions exist requesting different coverage probabilities.

234. GUM 2 9.3.3 Line 4 and

In the context of calibration certificates is We suggest to use only 95 % coverage probability, such as in

Doing so would be


of 90

EXAMPLE 2

statement that “coverage probability p such as 0.9, 0.95 or 0.99” and “A calibration certificate states that … for a coverage probability of 99 %”. The recommendation of ILAC states “The reported expanded uncertainty of measurement is stated as the standard uncertainty of measurement multiplied by the coverage factor k such that the coverage probability corresponds to approximately 95 %.” in the document ILAC-P14:01/2013 (ILAC Policy for Uncertainty in Calibration)

GUM-S1, especially in the context of the expanded uncertainty derived from the calibration certificate. The 95 % coverage probability is more realistic than 99 % or eventually 90 %.

restrictive.See response to 233.

235. MSL 7 9.3.3 Example 2

The note here seems odd, why in this particular example would one suspect that a Gaussian PDF not be suitable? It seems out of place in this section.

Consider deleting the note The text will be changed.

236. NMIA 3 9.3.4 First

paragraph

Calibration certificates from third parties very often do not provide veff.

Include advice on how to calculate u(x) when veff is not known

Accepted.

237. VSL 23 9.3.4 also table

2

At first view seems inconsistent with 11.1.3 (Up <> kp*u). It is probably not inconsistent, because the certificate is based on GUM2008. It may not be clear which GUM version has been used in practice, so this may be awkard. What if you have a certificate with U, kp and \nu based on new GUM? 9.3.3 only deals with normal distribution. It seems strange that you transform by default the t-distribution to a normal distribution. The note from point 9.3.3 may be promoted to a separate subsection

Change notation to Up_old and kp_old and write \sigma = Up_old/ kp_old. Split 9.3.4 in two subsections: one for certificates stating student-t (Up, kp, \nu) based on GUM2008 and one for certificates based on GUM201X. According to 12.2.3 Note 1 it is still possible to have certificates with student-t (Up, kp, \nu) under the new GUM.

Only certificates based on the current GUM might provide nu_eff. Certificates produced according to the proposed revised GUM would not provide nu_eff.

238. VSL 25 9.3.4 table 2

Avoid confusion Replace k_p by other symbol, e.g. k_p,old in the row on t-distribution, to make clear that this k_p is different from the k_p

As 237.


of 90

in 11.1.3 239. VSL 26

9.3.4 table 2 completeness Add row with available

knowledge (Up, kp, \nu) obtained with approach based on new GUM

As 237.

240. MSL 8 9.3.5 P1/L1

The sentence seems awkward with ‘given’ Remove the word ‘given’ Accepted.

241. Inmetro 19 9.3.6

Better reading and understanding by adding an example.

In accordance with the previous subclauses, we suggest the inclusion of a practical / real example.


242. LNE 48 9.3.6 at the end

of §

No example provided Add an example Under consideration by JCGM-WG1.

243. LNE 43 9.3.6 Table 2

The sinusoidal cycling is illustrated no more by an Arc sine law but with a derivative arc sine.

Replace Arc Sine (U-shaped ) by Derivative Arc Sine (U-Shaped)

Text to be clarified. Courbes Arcisnus et Dérivée d'arcsinus

-8

-6

-4

-2

0

2

4

6

8

-1.5 -1 -0.5 0 0.5 1 1.5

x

y

g = Arcsinus g' = Dérivée d'Arcsinus


of 90

244. MSL 9 9.3.7 P1/L1

The sentence seems awkward with ‘given’ Remove the word ‘given’ Accepted.

245. VSL 24 9.3.7 also table

2

At first view unrealistic and counterintuitive pdf. pdf Ex(1/x) for a length, temperature or pressure without uncertainty is strange. Highest probability at zero!

Give more justification for the usefulness and nice properties of this pdf, or remove this pdf completely.

Accepted.

246. LNE 49 9.3.7 at the end

of §

No example provided Add an example Under consideration by JCGM-WG1.

247. Inmetro 20 9.3.7

Better reading and understanding by adding an example and modifying notation.

Inclusion of an example as mentioned above. In addition, we also suggest changing the notation for the exponential to exp(1/x), as proposed on item 16 for Table 2.


248. PTB 22 9.3.8 Title

“Elicitation” is not self-explaining Change to “Elicitation of experts knowledge for the formation of PDFs”

Accepted.

249. CENAM 37 9.3.8

Log-normal PDF appears often in some situations, like particle size determinations by electron microscopy. A procedure to deal with it is to change the original variable x to log x, to obtain results under treatment as a Gaussian, and then going back to the original variable.

Address log-normal PDF by itself, or as an example of change of variables to proceed through an easier treatment.


250. Inmetro 21 9.3.8 Pages 21

to 23

Better reading and understanding by adding examples.

As the topic (elicitation) is a novelty in the draft, it would be appropriate to include practical examples in each subclause.


251. LNE 50 9.3.8.1 Table 3

table 2 and 3 are not in accordance when the knowledge is: best estimate x and standard uncertainty u(x). One suggest to use a Gaussian distribution while the other suggest to use the scaled and shifted t distribution

Ensure consistency or explain the difference


252. VSL 27 9.3.8.2

incomplete information give guidance how to calculate Under


of 90

alpha and beta, otherwise it won't be used

consideration by JCGM-WG1.

253. LNE 51 9.3.8.2

It is unclear how to calculate values for α and β

Add a practical example in this paragraph


254. Inmetro 22 9.3.8.2


We suggest to indicate references (and / or include a practical example) on how to estimate the parameters 𝛼𝛼 and𝛽𝛽for the Beta distribution.


255. LNE 52 9.3.8.2 Middle of

the paragraph

What is J-, L- ? Define these notations Under consideration by JCGM-WG1.

256. METAS 6 9.3.8.2 / 9.3.8.3

Instructions in the use of beta distribution and t- distribution in the elicitation process are very vague.

2 possibilities: 1) Give these two distributions less room in the guide and provide instead references on how to use these distributions in the elicitation process. 2) Give more detailed instructions, eventually examples.


257. CEM 9.3.8.3

When beta distribution is described, the way to obtain α and β from experimental data should be included. Otherwise the inclusion of this distribution is completely useless.


258. VSL 28 9.3.8.3

incomplete information give guidance on when to select a normal distribution and when t-distribution and how to select number of freedom. It looks very arbitrary now. Reliability of current GUM can be used?


259. LNE 53 9.3.8.3

Same problem of consistency as the one illustrated in table 2. It’s not clear either to

Harmonize or explain the difference

Under consideration


of 90

decide between a Gaussian or a Student distribution

by JCGM-WG1.

260. PTB 23 9.3.8.3 Note

Why does the choice make “no difference” although the length of the coverage interval can be affected?

Add a reference “(clause 3.4)” after “Guide”.


261. Inmetro 23 9.3.8.3 NOTE

Better reading and understanding. We suggest better explanation of the term “no difference”, avoiding confusion. There is no difference by using the methodology described by the GUM (LPU), but there will be a difference in the final value for coverage interval for the measurand.


262. LNE 54 9.3.8.4

In this subsection, only the symmetric triangular distribution is concerned. If one uses an asymmetric triangular distribution, then the best estimate is different from the midpoint

Change the title : “Symmetric triangular distribution”


263. LNE 55 9.3.8.4 Last

sentence

“A triangular distribution also arises as the convolution of two rectangular distributions with identical semi-widths.” One should specified that this sentence is correct when dealing with two rectangular distributions

Change the sentence accordingly : “A triangular distribution also arises as the convolution of two rectangular distributions with identical semi-widths”

The proposal is identical to the existing text.

264. Inmetro 24 9.3.9.1


Add a reference to the definition of ancillary statistics. Although it is a common concept in Bayesian inference, little is known about it in the metrology community, which can create confusion in understanding.


265. CMI 4 9.3.9.2 lines 5 - 7

The formulations in both sentences are confusing: stands Ei for one of the components or for just the error of indication ? (The plural in the second sentence).

To make the text more clear. Accepted.

266. CENAM 38 9.3.9.2

The user may become confused between the Include a note warning the Text to be


of 90

content of 7.2.5 and 9.3.9.2, since the former requires to make corrections due to systematic effects and not to include them as contributions to the MU, and the later allows assigning a contribution to the limited information due to conformity assessment results.

difference between 7.2.5 and 9.3.9.2.

clarified.

267. Inmetro 25 9.3.10.1

Better linking of the text. At the bottom of the text, include: "See example in clause 9.3.10.3."

Accepted.

268. PTB 24 9.3.10.3 Sentence

starting with “See also ...” after Eq. (20)

It is unclear here what formulae (38) and (39) are good for.

Add “for correlated quantities Q1,... ,QL.” after “in 10.4.3.3”

Accepted.

269. PTB 25 9.3.10.3 Sentence

starting with “It is also ...” after Eq. (20)

It is unclear how to handle this situation. Delete this sentence or give guidance.

Accepted.

270. CMI 5 9.3.10.3 Note 1, the

text after the formula (21)

Probably a typing error in u(x1, x2) – should it not be u(xi, xj) ?

To correct it. Accepted.

271. PTB 26 9.3.10.3 Note 1

It is not appropriate to define the correlation coefficients in a note.

Transfer note 1 to regular text. Accepted.

272. Inmetro 26 9.3.10.3 NOTE 1

and EXAMPLE

Better understanding by simplifying the example or modifying the text.

Initially we suggest changing the given example for a simpler one, which would be easier to understand. Otherwise, we suggest the following changes: (i) In Note 1, replace 𝑢𝑢(𝑥𝑥1, 𝑥𝑥2)for 𝑢𝑢(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗); (ii) replace the formula

(i)Accepted. (ii) The proposal is too general. We would also have to


of 90

𝐶𝐶𝐶𝐶𝐶𝐶(𝑋𝑋1 ± 𝑋𝑋2,𝑋𝑋3) = 𝐶𝐶𝐶𝐶𝐶𝐶 (𝑋𝑋1,𝑋𝑋3) ± 𝐶𝐶𝐶𝐶𝐶𝐶 (𝑋𝑋2,𝑋𝑋3) by the more general formula: 𝐶𝐶𝐶𝐶𝐶𝐶(𝑎𝑎𝑋𝑋1 ± 𝑏𝑏𝑋𝑋2, 𝑐𝑐𝑋𝑋3 ±𝑑𝑑𝑋𝑋4) = ⋯; (iii) by the end of the example, in the passage: "In this application, the two masses 𝑚𝑚1 and 𝑚𝑚2 are correlated input order quantities in the measurement model. See 10.4.3.1 and 10.4.3.1, "replace the last phrase"See 10.4.3.1 and 10.4.3.1"with"See 10.4.3.1 and 10.4.3.2"; (iv) by the end of example include: "Expressing 𝑚𝑚 = 2𝑚𝑚𝑠𝑠 − ∆𝑚𝑚1 −∆𝑚𝑚2, the input quantities of m are not correlated. See 9.3.10.1".

say that a, b, … are constants. (iii)Accepted. (iv) See 9.3.10.4

273. PTB 27 9.3.10.3 Example,

after Eq (22)

The relation between u(xi,xj) and Cov(Xi,Xj) is not mentioned.

Add „where u(…)=Cov(…)” Already in (24).

274. LNE 56 9.3.10.3 Example –

line 20

The reference 10.4.3.1 is repeated twice in the sentence

“See 10.4.3.1” Accepted.

275. PTB 28 9.3.10.3 Paragraph

after equation (23)

A verb is missing in the sentence “Applying the rule Cov(X1 ± X2, X3) = Cov(X1, X3) ± Cov(X2, X3),…”

Change to “Applying the rule Cov(X1 ± X2, X3) = Cov(X1, X3) ± Cov(X2, X3), leads to …”

Accepted.

276. PTB 29 9.3.10.3 Note 2

This text should not be a note. Change to example text. Accepted.

277. LNE 57 9.3.10.3 NOTE 2

A continuation of “EXAMPLE”. (“NOTE 2” follows “example” which follows “NOTE 1”) NOTE 2 is specific to the example and then its first note

Rename NOTE 2 in NOTE 1 Accepted.

278. MSL 10 10 Title

The title should be changed as the Try: “Estimating the measurand Accepted.


of 90

measurand cannot be evaluated and evaluating the standard uncertainty”

279. VSL 30 10

We have concerns about the requirement of validating the law of propagation of uncertainty for non-linear measurements model in that it may constitute a serious hurdle for many users of the GUM

Pay more attention to the fact that the readership of the GUM is not necessarily well educated in mathematics and statistics; provide more (detailed/specific) guidance on how to validate the law of propagation of uncertainty.


280. VNIIM 10 10. Evaluating the measurand and standard uncertainty

10.1 The last sentence states only standard uncertainties and the covariance is not mentioned. The first sentence mentions the covariance. It makes a wrong impression that this change somehow relates to linearization of the model.

The inaccurate wording should be corrected.

Accepted.

281. PTB 30 10.1 Last

sentence

Wrong, correlation may be needed. Substitute the sentence by “No explicit usage other than the best estimate, associated standard uncertainty and possible covariances are required of the PDFs for the input quantities.”

Accepted.

282. MSL 11 10.2 Title

The title should be changed as the measurand cannot be evaluated

Try: “Estimating the measurand” Accepted.

283. PTB 31 10.2 First

sentence

The reference to (10.1) is circular. Better refer to (10.3, 10.4)

Substitute “(see 10.1)” by “(see 10.3, 10.4)”.

Accepted.

284. PTB 32 10.2 Note

This note is unclear.

Start the Note by “Assume the input ...”

Accepted.

285. UNIIM 1 Sub Clause 10.2

Note Formula 27 (a, b)

The draft document considers an important question, related to two methods of evaluation of measurement results, which give identical estimates for linear model. But no clear conclusion is given on how to act for non-linear model of indirect measurements,

Numerical example should be given for a non-linear model of indirect measurements for this aspect.



of 90

when two estimates do not coincide. Only references to scientific articles are provided [1],[2],[3],[4].

286. NMIJ 20 10.2 Note

Two formulas (a) and (b) for calculation of the estimates of a measurand are presented, and the former is recommended. However, this recommendation is contrary to that in the current GUM (4.1.4).

Explain the reason for the change of recommended formula.


287. MSL 12 10.2 Last

sentence

The preferred choice in (27) is not identified Identify option (a) or (b) in equation (27)


288. Inmetro 27 10.2

Better reading and understanding. In the numbering of the equation (27), replace it by (27a) and include the numbering in the following equation (27b). In addition, replace the symbols𝜉𝜉 as commented before.


289. VNIIM 11 10.2

In the note, both estimates are given under number (27), so it is unclear which one authors recommend and in which cases.

The text, referring to the comparison of a) and b) estimates, should be clarified. A brief description of the benefits of a particular estimate should be given.


290. LNE 58 10.2 NOTE

The choice between the two expressions (formula 27 and the other) may depend on the measurement process :

• all the input quantities are measured together

and this operation is repeated n times (use the other formula)

• the input quantities are measured n times separately (use the formula 27)

Replace the sentence “In general, option (27) is to be preferred” by “The choice between the two expressions (formula 27 and the other) may depend on the measurement process :

• all the input quantities are measured together

and this operation is repeated n times (use the other formula)

• the input quantities are



of 90

measured n times separately (use the formula 27)

291. LNE 59 10.2 NOTE

Option (27): 27 is not an option reference it is an equation number. Options are a) and b)

Replace “option (27)” by “option a)”

Accepted.

292. VNIIM 12 10.3

It would be helpful to use an estimate of the remainder in the Taylor expansion to check the applicability of the linear approximation of the model.

It is advised to add to 10.3 an estimate of the remainder in the Taylor expansion to verify the adequacy of the linear approximation application for nonlinear measurement models.


293. LNE 60 10.3.1 First

paragraph

Very important statement, arrives too late in the document

Bring forward the first paragraph in Clause 3


294. MSL 13 10.3.1 P1/L5

It seems unwise to stipulate that the particular MC method of S1 should be used, when other ways of implementing the “Propagation of Distributions” may be available.

Consider re-wording so as to be less restrictive.

Accepted.

295. NPL 2 10.3.1 Figure 2

Ensure that Figures 2 is meaningful when the document is not printed in colour – most ISO documents do not contain colour.

Produce a suitable version of Figure 2 in grayscale for inclusion instead of the colour version.

Accepted.

296. PTB 33 10.3.1 Figure 2

The blue and red curves are optional – add this information.

Change the legend to “possible PDFs of X” or “assumed PDFs of X”

Accepted.

297. LNE 61 10.3.1 Figure 2

The figure is not clear enough Make two subfigures, one for each case of PDF for X


298. LNE 62 10.3.1 Figure 2

What is the unit of abscises? Why does X lies between 10 and 14?

Explain Accepted. Axis labelling will be removed.

299. LNE 63 10.3.1 Figure 2

The figure 2 gives only the explanation for a function of a single parameter

After this figure, precise that it is also practically possible to calculate the partial derivatives

Accepted.


of 90

of the model in order to verify that each of these are constant on the dispersion interval

300. VSL 31 10.3.1

unclear what 'described above' refers to add explanation Accepted.

301. PTB 34 10.3.2 First

sentence

Confirming the adequacy of the application of GUM is not optional (“may”) in cases of doubt.

Substitute “may” by “has to be”. Accepted.

302. CMI 6 10.3.2 4th line

10.4

Any references to the text downwards should ideally be avoided.

Such reorganization of all such material would introduce other problems.

303. NMIJ 21 10.3.2

When the validity of linear approximation of a nonlinear model is doubtful, the use of the Monte Carlo method described in the GUM-S1 is suggested. However, no criterion for necessity to use the MC method is given. As a consequence, the use of the MC method may be taken as mandatory whenever the model is nonlinear.

Give a simple criterion on whether the validity of linear approximation should be verified by the Monte Carlo method.

There is no such simple criterion.

304. Inmetro 28 10.3.2

Using Monte Carlo method only for verification makes sense if it is valid also for similar tests. Otherwise, it is best to use only the Monte Carlo method directly.

We suggest therefore including a note on the use of a simple technique to assess the linearization effect for the function and the standard uncertainty of each Xi. For example, the technique on which a perturbation of the measurement function 𝑌𝑌 = 𝜕𝜕(𝑋𝑋1, … ,𝑋𝑋𝑛𝑛) is used to check if the evaluation of uncertainty by Monte Carlo is necessary.


305. NMIJ 22 10.4

When the number of repetitions in a dominant "type A" uncertainty is small, the resultant

Explain what will happen when a dominant "type A" uncertainty is

9.2.4 considers this


of 90

coverage interval is unreliable with poor reproducibility, which is much the same as in the case of the current GUM.

based on a small number of repeated measurements. Also add a note on appropriate choice of the number of repeated measurements.

aspect.

306. INTI

10.4.1.1

When the model is stated as a computing algorithm (see 2.2.), rather than as a mathematical relationship, the sensitivity coefficients must be calculated numerically

To mention the possibility that the sensitivity coefficient are calculated numerically

Accepted.

307. PTB 36 10.4.1.2 2nd line

“the measurement model (1) (in 3.1)”: the model is given in 3.2, not in 3.1

Change to “(in 3.2)” Accepted.

308. PTB 35 10.4.2 title and

elsewhere

Unnecessary restriction to independent input quantities

Substitute “independent” by “uncorrelated” (or mention both) throughout 10.4.2.


309. INTI

10.4.2

Independence is a sufficient but not necessary condition to achieve null correlations

10.4.2 “ Uncorrelated Independent input quantities”. Also, the term independent should be changed several times in the paragraph

Unclear proposal.

310. VNIIM 13 10.4.2

In Example 2, reference should be made to 9/2/2 and 9.2.3 instead of 9.2.2.

To be corrected as suggested. Accepted.

311. VSL 32 10.4.2.1 Note 3

insufficient information add some words on how sensitivity coefficients can be calculated

Accepted.

312. LNE 64 10.4.2.1 NOTE 3

In the case of a computational or algebraically complicated measurement model, one would recommend also to apply GUM-S1, because of the risk of numerical errors in the calculation of derivatives and it is often more practical to generate a sample of the input quantities, which is used as input data of a computational model, in order to analyse the output sample

Complete NOTE 3 with “[…] measurement model. However, it is often more convenient to apply GUM-S1 in such cases.”

Accepted.

313. LNE 65 10. 4.2.1

Although both terminologies are correct, “Quadrature sum” is less used than

Replace “Quadrature sum” by “Quadratic sum”

In JCGM’s experience,


of 90

‘Quadratic sum” in various scientific communities

quadrature sum is more common.

314. PTB 37 10.4.2.2 1st line,

last but second word.

In the rest of the document “th” is not in superscript.

Change “ith” to “ith”. Accepted.

315. MSL 14 10.4.2.2 Example 2,

L5

Replace “indications” with “indication values”

The indications here are quantities.

316. LNE 66 10.4.2.2 Example 2,

Line 7

Reference 9.2.2 is incorrect Change to 9.2.3 Accepted.

317. NMIA 4 10.4.2.2 EXAMPLE

2

In our extensive experience, the errors of digital voltammeters do not follow rectangular distribution. It is quite possible that the measured error is outside of manufacturers’ specification. Manufacturers use a large number of instruments to arrive at specifications, pointing to a Gaussian distribution. Industrial users in Australia presently use Gaussian distribution to describe this uncertainty component.

Change to Gaussian distribution.

The example is only illustrative. To use another PDF would require further knowledge.

318. Inmetro 29 10.4.2.3 Line 2

Better reading and understanding. We suggest replacing the excerpt "application of the formula (32) to expression (34)" to "the application of formula (31) to expression (34)".

Accepted.

319. MSL 15 10.4.2.3 Note 2, L1

Replace “gives” with “give” Accepted.

320. VSL 33 10.4.2.3

introduce symbol u_rel uncertainties u_rel: Accepted.


of 90

321. LNE 67 10.4.2.3 Formula

(35)

‘Gives a relation between relative standard uncertainties’, not uncertainties but variances

Gives a relation between relative variances

Current text is preferred.

322. LNE 68 10.4.3.1 Example,

Line 5

“The standard measurement uncertainty associated with estimate … is obtained by application of formula (36)”. Formula (36) gives a variance, not a standard uncertainty.

Replace “standard measurement uncertainty” with either “squared standard measurement uncertainty” or “variance”


323. LNE 69 10.4.3.1 Example

“Similar cases occur frequently…(see note 2 to 9.3.10.3)” No similar cases are indicated in the note 2 of 9.3.10.3.

Bring clarification Accepted. Reference to be deleted.

324. LNE 70 10.4.3.2

“The standard uncertainty … is given by” The expression below is a variance. See similar comment above


325. LNE 71 10.4.3.2 Line 2

En is very often defined as : (y1-y2)/ √(u2y1+u2y2) (see, for example, § B3 in CEI 17043:2010). It could be useful to add the reference of the definition

Add the reference of definition Text to be clarified.

326. NMIJ 23 10.4.3.2 Paragraph

No. 1/Line No. 2

The symbol for the normalized error, En (= (y1 − y2) /u(y1 − y2)), is not appropriate, because it is used to indicate half the value of this quantity in ISO 17043 and ISO 13528.

Use a symbol other than "En" to represent (y1 − y2) /u(y1 − y2).

Text to be clarified.

327. Inmetro 30 10.4.3.3 Last

paragraph

Better reading and understanding. Replace the excerpt “when the qL…” to “when the ql…”.

The text is already written that way.

328. LNE 73 11.1 or 11.2

Why has the Central Limit Theorem disappeared from the proposal? This theorem is of a key importance in determining the expanded uncertainty

Explain the importance of TCL in this section or the previous one

The CLT applies only in particular circumstances.

329. PTB 38 11.1.1 Sentence

starting with “Such

This is not true generally (e.g. credible intervals).

Substitute the sentence by “Such an interval is termed a coverage interval here and ...”

Accepted.


of 90

an interval ...”

330. MSL 16 11.1.2 Note 1

The first sentence repeats what is already in the section

Delete the first sentence Accepted.

331. LNE 72 11.1.2 NOTE 1

The case of PDFs having more than one peak could be problematic. In this case what is the best estimate? Potentially, the best estimate could lie outside of the coverage interval in the case of a double-peaked PDF, with equal probability for both peaks. Moreover, this statement is not consistent with GUM-S1 that only deals with joint coverage intervals [ylow;yhigh]

Either suggest to remove the sentence beginning with “Exceptionally, the shortest …” or explain in more details potential issues arising in such cases


332. Inmetro 31 11.1.3

Better reading and understanding. We suggest in this subclause to give a simple explanation about the coverage factor as a quantile of the distribution in question. This explanation about the nomenclature (common for statistics) can facilitate understanding and avoid confusion between the two terms.

Unclear proposal.

333. PTB 39 11.1.4 Last

sentence

It is not clear what this means. Delete sentence Accepted.

334. INTI

11.2

Distribution free intervals are a useful option only when the target uncertainty is large enough. If not, some information on the output pdf has to be considered and shorter intervals will be got. However, some readers of the draft have misunderstood this, believing that conservative intervals are proposed as a general rule. Therefore the new GUM would increase the final uncertainties in a broad variety of cases

To clarify very well when conservative intervals are reasonable only when target uncertainties are large enough

Accepted. Text will be added.


of 90


of Metrolo

gy”, Ukraine

5 11.2 -

In Clause 11.2. states that in case of absence of information about the distribution of the measurand for find the coverage factor is necessary to use an expression of the Chebyshev inequality. This can lead to incorrect estimates of the expanded uncertainty.

The Clause 11.2 to exclude The expression so obtained is not incorrect, but, as stated, generally conservative.

336. NMIJ 24 11.2, 11.3, 11.4

Coverage factors are given with three significant digits, which might cause misunderstanding that standard uncertainties have accuracy with the same significant digits. It is inappropriate to expect excessive accuracy in uncertainties.

Give coverage factors in at most two significant digits. In particular, replace 1.96 with "approximately 2".

Intermediate values in a computation are often held to more digits.

337. LNE 74 11.2.1

The third sentence is not clear : “However, .. and thus valid independent of the shape of the PDF”

Change to : “ ..and thus are valid independent of the shape of the PDF”

Accepted.

338. CMI 7 11.2.2,

11.2.3

It might be advisable to derive both formulas in an annex (not a strong comment – Chebyshev inequality is well known and the derivation of the Gauss inequality might be too elaborate).

To consider the proposal. JCGM decided to keep the document as short as possible.

339. LNE 75 11.2.2

11.2.3

11.2.4

The use of Chebyshev inequality will be very limited in practice. In general, either people have good confidence in a Gaussian PDF for the measurand (and they can switch to the case where Y can be regarded as Gaussian (11.3.1)) or they have no idea (and they should apply the GUM-S1). The results provided by Gauss and Chebyshev inequalities are too much conservative and would lead to too many changes in decision making, calibration certificates and conformity assessment. Such a choice to provide a coverage interval is then unrealistic for many

Remove 11.2 Text will be added to clarify.


of 90

users in different disciplines 340. CEM 3

11.3 Is central limit theorem no longer valid? CLT applies

(only) in particular circumstances.

341. CENAM 39 11.3.1

There is some lack of clarity by including just one example for 95% coverage probability.

Include a note emphasizing the coverage factor for a Gaussian distribution or reference to section 9.3.3

1.96 is illustrative, but “popular”.

342. NMIJ 25 11.3.1

Because a "Type A" standard uncertainty is already enlarged by the factor √(n-1)/(n-3), the use of k = 2 for 95 % coverage interval gives an appropriate (i.e. conservative) estimate in most practical cases. In fact, this holds true for t-distributions (including Gaussians); uniform, U-shaped, triangular, and trapezoidal distributions; and any combined distributions of these PDFs. Considering that 95 % has been the primary choice as the coverage probability, it is desirable to emphasize the usability of k = 2 for 95 % coverage factor.

Add a note in 11.3.1 that the use of k = 1.96 (≅ 2) gives a conservative estimate of 95 % coverage interval for a wide variety of practical situations.


343. INTI

11.3.1

The Welch-Satterthwaitte formula became a much extended practice since the publication of the GUM in 1995. It will not be discarded by the users without a convincing explanation of the reasons to do that

It may be necessary to include a convincing comment deprecating the use of the Welch-Satterthwaitte formula, and the reasons for that

Accepted. Further details of the benefits of not using nu_eff will be included.

344. METAS 5 11.3.2

“…can be extremely difficult to obtain” seems a bit negative. From experience it is known that the linearization and Gaussian approximation work in most practical cases very well. “In all cases of doubt the Monte Carlo

“Linearization and the assumption that the PDF for Y can be regarded as Gaussian are in most practical cases good approximations. In cases of doubts the careful application of the Monte Carlo procedure can

Accepted.


of 90

procedure can be applied…” seems to be too optimistic, because MC can cause problems too if not carefully applied.

be used to obtain …”

345. NMIJ 26 11.3.2

It is described that the Monte Carlo procedure "can" be applied to confirm the adequacy of linearization and the type of PDF. It is unclear whether any validation methods other than the Monte Carlo method are available.

If using the Monte Carlo method is required for the validity check, "can" should be replaced by "should". However, it should be recognized that utilizing the MC method is not practical for the majority of people who evaluate calibration uncertainty.

“Can” is retained because “should” might rule out other approaches. Such approaches include analytical methods (clause 11.4) when they can be applied.

346. LNE 76 11.3.3

Coverage intervals may include (as a result of calculation) infeasible values. No recommendations are given to eliminate values which would be outside the permissible interval. Examples are given (see NOTE in 11.3.3 and EXAMPLE in 11.4.2) but no guidance is provided. Another example: the reflection coefficient measured in RF (radio frequency) metrology should be within [0,1]

Add recommendations for the expression of ‘realistic” coverage interval

This is also problematic for the existing gum framework. For a much larger class of problems, MC applies. A way to guarantee feasibility in all cases is to carry out a full Bayesian evaluation. A Bayesian Supplement is planned that will cover this.


of 90

347. CENAM 40 11.4

This section seems to be rarely used in practice.

This could be moved into an appendix

It is used by some NMIs and JCGM wishes to promote it more widely.

348. PTB 40 11.4.1 1st line

η is not explained. Add a reference “(see definition 5.3)” similar to the reference to 5.2 for G.

See Glossary.

349. CENAM 41 11.4.1

The expression “Analytical methods to obtain the PDF gY (_) for Y are ideal because a coverage interval can be obtained mathematically from the PDF.” Seems to suggest that numerical methods are not mathematical. Clarity is the aim, not a discussion on semantics.

Use another term replacing “mathematical” in this statement.

Text to be clarified.

350. CMI 8 11.4.2 Formula

(43)

No derivation of the rather important formula is available in the document. To buy a book to get an access to it is not very user-friendly.

To give the derivation in an annex.

JCGM-WG1 preferred to keep the document as short as possible.

351. PTB 41 11.4.2 Example

This is a little bit too short. Please give the ingredients for (43) explicitly for this example.


352. CMI 9 11.4.4

The concept and the example are not well explained, any benefit of it here is questionable.

To consider it. Under consideration by JCGM-WG1.

353. NPL 3 11.4.4 Figure 3

Ensure that Figures 3 is meaningful when the document is not printed in colour – most ISO documents do not contain colour.

Produce a suitable version of Figure 3 in grayscale for inclusion instead of the colour version.

Accepted.

354. PTB 42 11.4.4 Figure 3

A green line is missing on the right hand side. Add the green line. Accepted.


of 90

355. PTB 44 11.4.5 Example

s and s1 are not defined. Define s and s1. Accepted.

356. VSL 34 11.4.5 Example

typo s -> s_1 Accepted.

357. NMIJ 27 11.4.5 EXAMPLE

The expression (sx2)2/n is probably not correct. It should be (s1x2)2/n.

Correct (sx2)2/n as (s1x2)2/n. Accepted.

358. Inmetro 32 11.4.5 EXAMPLE

Better reading and understanding. We suggest some modifications to the example: (i) replacing "are given by expressions (9) to (11) in 9.2" for "are given by expressions (8) to (11) in 9.2"; (ii) replace: "𝑡𝑡𝑛𝑛−1(𝑥𝑥1𝑥𝑥2, +𝑥𝑥3, (𝑠𝑠𝑥𝑥2)2/𝑛𝑛" with "𝑡𝑡𝑛𝑛−1(𝑥𝑥1𝑥𝑥2, +𝑥𝑥3, (𝑠𝑠1𝑥𝑥2)2/𝑛𝑛".


359. VNIIM 14 11. Determining a coverage interval …

11.4.5 In example 11.4, to calculate the coverage interval instead of the coverage factor kp the argument of t-distribution is used, which contradicts the general provisions of paragraph 11.1.3 (coverage factor kp is a factor, which is multiplied by the standard uncertainty u, not by the sample standard deviation of the average). Besides, the argument of t-distribution in this case should correspond to the level of 100 (1+p)/2 %.

It is proposed to: − Specify correctly the

coverage interval using the expanded uncertainty:

𝑢𝑢(𝑦𝑦) = |𝑥𝑥2|𝑢𝑢(𝑥𝑥1) = |𝑥𝑥2|�𝑛𝑛−1𝑛𝑛−3

𝑆𝑆1√𝑛𝑛

𝑦𝑦 ± 𝑘𝑘𝑝𝑝𝑢𝑢(𝑦𝑦), where the coverage factor kp can be approximately equal to 2; or, − Or to give a formula for the

coverage interval without using the coverage factor (expanded uncertainty) and to clarify the notations.


360. PTB 43 11.4.5 Last

sentence

It is not evident why to use the argument of the t-distribution corresponding to the probability 100(1–p)/2% .

Substitute by “100(1+p)/2%” or “100(1 - (1-p)/2)”.


361. INTI 12.2.1

In almost all practical cases, the expanded 12.2.1.1 The following items Text will be


of 90

uncertainty is used instead the standard uncertainty for expressing measurement results. So, both options should be considered at the same level in the text

should always be given: a) The specification of the measurand (output quantity) Y and b1) The best estimate y of Y and the associated standard uncertainty u(y) or b2) A coverage interval for Y for a stated coverage probability and items in 12.3 are usually preferred to option b

modified.

362. METAS 7 12.2.1.1

According to this clause, it will be acceptable to express a measurement result only by the estimate and its standard uncertainty. This is less information than the current practice, where a coverage interval and an indication, on how this was obtained (coverage factor and PDF assumed to obtain the coverage factor) were given. For further use of the measurement uncertainty, e.g. as a contribution to another uncertainty or for the assessment of conformity with specification, more information is needed.

The statement of the coverage interval and the associated coverage probability should be an integral component in the statement of a measurement result.

Accepted. Text will be modified.

363. LNE 77 12.2.1.2

The reference of ISO 80000-1:2009 has been changed by ISO

Replace with : ISO 80000-1:2013

Accepted.

364. LNE 78 12.2.1.2

Compared to JCGM 100:2008, there is no longer a recommendation for rounding up the uncertainty in JGCM 100:201X CD. This change could be specified in a note for example In the previous GUM, it was suggested to round up uncertainties “in reporting final results, it may sometimes be appropriate to round uncertainties up rather than to the nearest digit.” In this new proposal, the rounding is done according to the standard

Include in note 1 a reminder of what was originally recommended in JCGM 100:2008.

Accepted. Rounding will be re-considered.


of 90

ISO 80000-1:2013. What is the need for this change? For many people, it is safer to round up their uncertainty value

365. PTB 45 12.2.1.3

This clause is about reporting on the input – not the output.

Move this clause to 12.3. This clause aims at giving guidance also in the multivariate case. Text will be make this explicit.

366. LNE 79 12.2.1.3

It is recommended here to provide correlation coefficients with 2 significant figures, based on 1-|r12|. Does this guarantee that the truncated covariance matrix will be positive definite?

If the positive definiteness of the covariance matrix is guaranteed, it would be interesting to mention it explicitly Otherwise, another truncation procedure should be provided/recommanded.

The matter is intricate and the existing text (slightly modified) is a reasonable compromise.

367. MSL 17 12.2.1.3 Example

the covariance value is incorrect the intended value is probably u(x_1,x_2)= -0.092 124 8

Accepted.

368. PTB 46 12.2.3 a)

Add the value of kp Change to “… modelled by a Gaussian distribution, i.e. kp = 1.96.

Accepted.


of Metrolo

gy”, Ukraine

6 12.2 12.2.3.

In Clause 12.2.3 examples of recording the results of measurements in sub clauses a) and c) do not meet the stated objectives.

The examples of recording the results of measurements in sub clauses a) and c) should be changed the places.

Unclear proposal.

370. Inmetro 33 12.2.3 Item c)

Better reading and understanding. We suggest replacing the words "assuming a symmetric PDF for mS" by "assuming the symmetric and single-peaked PDF for mS".

Accepted.

371. Inmetro 34 12.2.3 NOTA 3

We suggest the guide to inform the number of decimal digits to use when expressing the

Accepted.


of 90

percentiles p1 and p2. 372. PTB 47

12.2.3 Note 5 Note 5 should not be a note but main text. Transfer note 5 to main text. Accepted.

373. MSL 18 References

Many bibliographic references are not cited from the text

Delete the following references: [3],[4],[6],[9],[10],[12],[13],[14],[15],[25],[29],[34],[44]

JCGM-WG1 checked and found all references cited in the text.

374. LNE 80 Glossary p.39

The hat notation for the best estimates of a quantity is used in many places of the text without reference to its definition is section 6.3

The hat notation should be defined in the Glossary


375. Inmetro 35 Glossary

Better reading and understanding. In line with the suggestion to subclause 7.5.1, change the word “dummy”, since this classification may lead to confusion with the use of the same word to refer to categorical variables of type 1 or 0.


References [for INMETRO comments] [1] BIPM, IEC, I. ILAC, ISO, IUPAC, IUPAP, and OIML (2008a). Evaluation of measurement data – Guide to the expression of uncertainty in measurement. Technical report, Joint Committee for Guides in Metrology, JCGM 100:2008. [2] BIPM, IEC, I. ILAC, ISO, IUPAC, IUPAP, and OIML (2008b). Evaluation of measurement data – Supplement 1 to the "Guide to the expression of uncertainty in measurement--Propagation of distributions using a Monte Carlo method. Technical report, Joint Committee for Guides in Metrology, JCGM 101:2008. [3] BIPM, IEC, I. ILAC, ISO, IUPAC, IUPAP, and OIML (2009). Evaluation of measurement data – An introduction to the "Guide to the expression of uncertainty in measurement" and related documents. Technical report, Joint Committee for Guides in Metrology, JCGM 104:2009. [4] BIPM, IEC, I. ILAC, ISO, IUPAC, IUPAP, and OIML (2011). Evaluation of measurement data – Supplement 2 to the "Guide to the expression of uncertainty in measurement--Extension to any number of output quantities. Technical report, Joint Committee forGuides in Metrology, JCGM 102:2011. [5] BIPM, IEC, I. ILAC, ISO, IUPAC, IUPAP, and OIML (2012a). Evaluation of measurement data – The role of measurement uncertainty in conformity assessment. Technical report, Joint Committee for Guides in Metrology, JCGM 106:2012.


of 90

[6] BIPM, IEC, I. ILAC, ISO, IUPAC, IUPAP, and OIML (2012b). International Vocabulary of Metrology: Basic and General Concepts and Associated Terms. Technical report, Joint Committee for Guides in Metrology, JCGM 200:2012. [7] Casella, G. and R. Berger (2002). Statistical Inference. Duxbury advanced series in statistics and decision sciences. Thomson Learning. [8] Ehrlich, C. (2014). Terminological aspects of the Guide to the Expression of Uncertainty in Measurement (GUM).Metrologia 51 (4), S145. [9] Gelman, A., J. B. Carlin, H. S. Stern, and D. B. Rubin (2003). Bayesian Data Analysis.Chapman and Hall/CRC. [10] Possolo, A. and B. Toman (2007). Assessment of measurement uncertainty via observation equations.Metrologia 44 (6), 464.

jcgm/wg1 template for reporting the comments on supplement 1 · transition between old and new...

Documents