G. CowanRHUL Physics Comment on use of LR for limits page 1

Comment on definition of likelihood ratio for limits

ATLAS Statistics Forum

CERN, 2 September, 2009

Glen CowanPhysics DepartmentRoyal Holloway, University of Londong.cowan@rhul.ac.ukwww.pp.rhul.ac.uk/~cowan

G. CowanRHUL Physics Comment on use of LR for limits page 2

IntroductionAt the Statistics Forum on 8.7.09 N. Andari presented a study (Orsay & Wisconsin) showing that a modified definition for the likelihood ratio leads to a sampling distribution that accurately follows the half-chi-square distribution.

This offered the possibility to increase greatly the ease and accuracywith which we can compute exclusion limits, even for small samples.

At the time some (at least GC and EG) did not fully understandhow this could work, so we thought through a simple example.

Conclusions agree with approach of Andari et al.

Purpose of present talk is to present this example; see also attachednote.

G. CowanRHUL Physics Comment on use of LR for limits page 3

The simple problemSuppose outcome of measurement is Gaussian distributed x with expectation value and variance

s and b are contributions from signal and background,take as a known constant; is strength parameter.

The likelihood function for the parameter of interest is

Suppose goal is to set an upper limit on given a measurement x.

G. CowanRHUL Physics Comment on use of LR for limits page 4

The likelihood ratioTo test a value of , construct likelihood ratio:

Suppose on physical grounds should be positive, thenthe maximum of L() from the allowed range of is from

Usually use logarithmic equivalent 2 ln ():

G. CowanRHUL Physics Comment on use of LR for limits page 5

Likelihood ratio for upper limitFor an upper limit on one uses the test statistic

which, putting together the ingredients, becomes

G. CowanRHUL Physics Comment on use of LR for limits page 6

p-value for exclusionTo quantify level of agreement between observed x andhypothesized , calculate p-value

Note if x ~ Gaussian(s+b, ), then the quantity

follows a chi-square pdf for 1 d.o.f.

But the distribution of q is more complicated (not chi-square),

95% CL upper limit on is value for which p = 0.05.

G. CowanRHUL Physics Comment on use of LR for limits page 7

Likelihood ratio without constraint on ̂Andari et al. propose to define an unphysical estimator

which goes negative if x < b.

Using this define then a corresponding test statistic for upperlimits:

G. CowanRHUL Physics Comment on use of LR for limits page 8

Exclusion significance from q′From the definition of q′ one can see its pdf must be a half-chi-square distribution, i.e., a delta function at zero when x > s + b, and a chi-square pdf for x ≤ s + b.

Therefore (see CSC note), the significance from an observed valueq′ is given by the simple relation

p-value of 0.05 corresponds to Z = 1.64.

G. CowanRHUL Physics Comment on use of LR for limits page 9

Comparison of test variables

Both q and q′ are shown here as a function of x for =1, s=10, b=20, 2=20.

Note they are equal for b ≤ x ≤ s + b.

G. CowanRHUL Physics Comment on use of LR for limits page 10

Equivalence of q and q′It is easy to see that the two test variables q and q′ aremonotonically related:

and therefore they represent equivalent tests of .

G. CowanRHUL Physics Comment on use of LR for limits page 11

Relation to study by Andari et al. (Eilam)Andari et al. presented the following table of the fraction oftoy experiments with values of the test statistic below certainlevels:

Chi-square and “exact” (counting) formulae give same fractionsfor “median”, but not median +1, +2. This is because in thisexample it corresponds to having x < b, i.e., this where q and q′ are different.

But for example, median 1 or 2 would correspond to the regionwhere q and q′ are equal, and the two methods there would agree.

G. CowanRHUL Physics Comment on use of LR for limits page 12

Conclusion on q′Both test variables, q and q′ , give equivalent tests, becauseof their monotonic relation.

If one were to work out (with difficulty) the exact sampling pdfof q, and to compute from it the p-value, and from it thesignificance Z, then it would be the same as from the simple formula Z = √ q′ using the same value of the observation x.

We still regard only positive as physical, but allow its estimator to go negative effectively as a mathematical trick to get the desired p-value.

Also easy to show that this likelihood ratio gives a test equivalentto the ratio used for the LEP analyses (see attached note):

G. CowanRHUL Physics Comment on use of LR for limits page 13

Extra slides

G. CowanRHUL Physics Comment on use of LR for limits page 14

Example from CSC bookE.g. H → from the CSC combination chapter used thestatistic q (as did all other channels).

MC studies show that distribution of q departs significantly from half-chi-square form.

If we had used q′, the agreement with half-chi-square wouldbe much closer, even for low luminosity.