results of combination higgs toy combination, within and across experiments, with roostats
DESCRIPTION
Results of combination Higgs toy combination, within and across experiments, with RooStats. Gr égory Schott Institute for Experimental Nuclear Physics of the Karlsruhe Institute of Technology on behalf of the ATLAS-CMS Higgs groups and statistics forum. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/1.jpg)
Results of combination Higgs toy Results of combination Higgs toy combination, within and across combination, within and across
experiments, with RooStatsexperiments, with RooStats
GrGrégory Schottégory Schott
Institute for Experimental Nuclear Physicsof the Karlsruhe Institute of Technology
on behalf of the ATLAS-CMS Higgs groups and statistics forum
ATLAS-CMS statistics meeting, 5th July 2010
![Page 2: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/2.jpg)
2Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Introduction I will present the results of ATLAS, CMS and the combination
summary of the results obtained by Kyle Cranmer and myself thanks to Kyle for the FC, PL and some Hybrid results in case of PL we both get results that of course agree very well
This is on-going work; while a large part of the results are available, not everything has been done yet and not everything has been validated yet
Toy study (made with mockup numbers), H→WW at 160 GeV with 1fb-1
don't draw physics conclusion of the results
Significance and upper limit computed with RooStats (for a given data) different approaches tested as recommanded by the statistics forums
Validation of RooStats (ROOT 5.27.04, June 2009) Conclusion and discussions
![Page 3: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/3.jpg)
3Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
The combined model Combination of H→WW→ee, , e analyses Parameter of interest: ratio of cross-sections, r = = / SM
CMS analysis: 3 observables, 37 nuisance parameters Lognormal distribution of nuisance parameters Uncertainty on control region measurement in the systematics
ATLAS analysis: 9 observables, 12 nuisance parameters Truncated gaussian distribution of nuisance parameters Control regions measurements included in the analysis
Combined analysis: common parameter interest (thus, 100% correlated) no other correlation (yet) across the experiments
Concretely, all the information for the combination is shared via 2 ROOT files (for ATLAS & CMS) containing RooFit/RooStats workspaces
graphical representation of the objects in the combined
model
![Page 4: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/4.jpg)
4Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Toy-data Data used for significance:
generated from the model in the hypothesis that signal is present
Data used for exclusion: generated from the model in the background-only hypothesis
Check the backups if you're interested by the number of observed events assumed (or the likelihood functions)
![Page 5: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/5.jpg)
5Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Profiled likelihood results
significance: reasonably good estimate if large enough
95% CL UL taken for -log (r) = 1.921 assumption of Wilks's asymptotia that's not always valid!
with systematics
r UL
ATLAS 0 0.79CMS 0 0.28
COMBI 0 0.25
with systematics
r signif.
ATLAS 1.13 2.70CMS 0.98 4.89
COMBI 1.02 5.58
^
^
![Page 6: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/6.jpg)
6Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Hybrid Frequentist-Bayesian
run for the 3 test statisticsmentionned in the previoustalk
need to study the advantageand inconvenient of the 3approaches
test statistics significance (no syst.)
significance (with syst.)
ATLASQLEP
QTEV
()
3.78-
3.07 ± 0.012.8 ± 0.1
-
CMSQLEP
QTEV
()
6.22 ± 0.02--
4.77 ± 0.02> 4.6
4.3 ± 0.1
COMBIQLEP
QTEV
()
---
> 4.6> 3.5
-
CMS, 477k toys
QLEP()QTEV
B-toysSB-toys
data
![Page 7: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/7.jpg)
7Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
CMS model with 'LandS'
Hybrid test statistics distributions
computing the p-value for significancein this approach is challenging:
speed improvements would be useful or use importance sampling techniques
CMS distribution (and results previousslide) made with a RooFit-independenttool
QLEPQLEP
![Page 8: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/8.jpg)
8Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Summary of upper limits
technique test stat rule sampling UL ATLAS UL CMS UL COMBIFeldman-Cousins (no
syst.)() CLS+B
toys 0.69 ± 0.05 - -
Profile LR (Wilks) () CLS+Basymptotic 0.79 0.28 0.25
Feldman-Cousins++ () CLS+Btoys 0.78 ± 0.05 0.26 ± 0.02 0.23 ± 0.02
Hybrid QLEP CLStoys ~ 0.68 0.29 ± 0.03
(LandS)-
Hybrid QLEP CLS+Btoys ~ 0.61 - -
Bayesian n/a, flat prior on r MCMC* 0.72 0.31 0.28
ATLAS
r
post
erio
r (r |
dat
a)
CMS
with BAT [ Caldwell, Kollar, Kroeninger, Comp.Phys.Com. 180, 2197 (2009) ]
95% CL UL
95% CL upper limits: results with systematics (except if indicated otherwise)
![Page 9: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/9.jpg)
9Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Validations Some validation of the analysis performed with independent software
(LandS, M. Chen, CMS) Since 'LandS' was developped specifically for this analysis, it is
faster than RooStats which is a general tool showing that RooStats still need to improve
table of compared results: we still need to work on the missing parts
PL and toy-MC results using an earlier version of the ATLAS model were validated with an independent code (Hao Liu, et. al.)
further on-going work on validating Bayesian limits: S. Schmitz (CMS) currently using BAT, RooStats providing another MCMCCalculator
![Page 10: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/10.jpg)
10Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Conclusion and outlook A good start, work is on-going on
Lessons learned: we achieved in a short time scale to combine the Atlas and
CMS analyses -> proof of principle that we can do that proof of principle we can compare different statistical methods
to one another still some weaknesses identified that RooStats need to improve we also need to continue the validations of RooStats
still need to understand those results discussion has started on statistical methods we want to use in common
proper treatment of correlations across experiments will require planning and thoughtful parametrization of systematics
activity will continue (other masses, other channels, more thorough correlation studies, ...)
![Page 11: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/11.jpg)
11Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
End of talk−
Backup slides
![Page 12: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/12.jpg)
12Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Toy-data Data used for significance:
generated from the model in the hypothesis that signal is present
Data used for exclusion: generated from the model in the background-only hypothesis Observables:
1) nobs_bin1 = 4 2) nobs_bin2 = 2 3) nobs_bin3 = 3 4) obs_s_em_0j = 19 5) obs_ww_em_0j = 63 6) obs_tt_em_0j = 3814 7) obs_wj_em_0j = 123 8) obs_s_ee_0j = 6 9) obs_ww_ee_0j = 14 10) obs_wj_ee_0j = 53 11) obs_s_mm_0j = 16 12) obs_ww_mm_0j = 26
Observables: 1) nobs_bin1 = 15 2) nobs_bin2 = 7 3) nobs_bin3 = 13 4) obs_s_em_0j = 36 5) obs_ww_em_0j = 52 6) obs_tt_em_0j = 3650 7) obs_wj_em_0j = 143 8) obs_s_ee_0j = 9 9) obs_ww_ee_0j = 5 10) obs_wj_ee_0j = 49 11) obs_s_mm_0j = 18 12) obs_ww_mm_0j = 28
![Page 13: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/13.jpg)
13Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
slide from K. Cranmer, 01.07.2010
ATLAS likelihood model
![Page 14: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/14.jpg)
14Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
CMS likelihood model
slide from A. Korytov, 24.06.2010
![Page 15: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/15.jpg)
15Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Profile likelihood (zoom)
![Page 16: Results of combination Higgs toy combination, within and across experiments, with RooStats](https://reader035.vdocuments.net/reader035/viewer/2022062521/568135bc550346895d9d1fff/html5/thumbnails/16.jpg)
16Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010
Bayesian upper limits
r
posterior (r | data)ATLAS
posterior (r | data)COMBI
r