analysis of astrophysics and particle physics data using...
TRANSCRIPT
![Page 1: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/1.jpg)
Analysis of Astrophysics and Particle Physics Data
using Optimal Segmentation
Space Science DivisionNASA Ames Research Center
Santa Cruz Institute for Particle PhysicsMay 17, 2005
![Page 2: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/2.jpg)
Outline
Goal: Detect/Characterize Local StructuresData CellsPiecewise Constant ModelsFitness FunctionsOptimizationError analysisInterpretationExtension to Higher Dimensions
![Page 3: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/3.jpg)
The Main Goal is to Detect and Characterize Local Structures
Background level
![Page 4: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/4.jpg)
From Data to Astronomical Goals
Data
Intermediate product(estimate of signal, image, density …)
End goalEstimate scientifically relevant quantities
![Page 5: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/5.jpg)
Smoothing and BinningOld views: the best (only) way to reduce noise is to smooth the data
the best (only) way to deal with point data is to use bins
New philosophy: smoothing and binning should be avoided because they ...discard informationdegrade resolutionintroduce dependence on parameters:
degree of smoothingbin size and location
Wavelet Denoising (Donoho, Johnstone) multiscale; no explicit smoothingAdaptive Kernel Smoothing
Optimal Segmentation (e.g. Bayesian Blocks) Omni-scale -- uses neither explicit smoothing nor pre-defined binning
![Page 6: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/6.jpg)
Data: Measurements Distributed in a Data Space
Independent variable (data space)e.g. time, position, wavelength, …
Dependent variablee.g. event locations, counts-in-bins, measurements, …
Examples: time series, spectraimages, photon mapsredshift surveyshigher dimensional data
![Page 7: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/7.jpg)
DATA CELLS: Definition
data space: set of all allowed values of the independent variable
data cell: a data structure representing an individual measurement
For a segmented model, the cells must contain all informationneeded to compute the model cost function.
The data cells typically:are in one-to-one correspondence to the measurementspartition the entire data space (no gaps or overlap)contain information on adjacency to other cells
… but any of these conditions may be violated.
![Page 8: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/8.jpg)
Simple Example of 1D Data Cells and Blocks
![Page 9: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/9.jpg)
Fitness Functions
Block likelihood = product of likelihoods of its cells
Block Likelihood depends onN = The Number of Events in the BlockM = The Size of the Block
Model likelihood = product of likelihoods of its blocks
Remove the dependence on the block event rates:Marginalize, orMaximize the Likelihood
Adopt prior distribution for Nb, the number of blocks. (Parameter of this distribution acts like a smoothing parameter.)
Take log to yield an additive fitness function.
![Page 10: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/10.jpg)
The Optimiserbest = []; last = [];for R = 1:num_cells
[ best(R), last(R) ] = max( [0 best] + fitness( cumsum( data_cells(1:R, :) ) );
if first > 0 & last(R) > first % Option: trigger on first significant blockchangepoints = last(R); return
end
end
% Now locate all the changepointsindex = last( num_cells );changepoints = [];
while index > 1changepoints = [ index changepoints ];index = last( index - 1 );
endDo not use at home: a few details omitted!
![Page 11: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/11.jpg)
For many iterations:Randomly select N of the observed events with replacementAnalyze this sample just as if it were real data
Compute mean and variance of the bootstrap samples
Bias = result for real data – bootstrap meanRMS error derived from bootstrap variance
Caveat: The real data does not have the repeated events in bootstrap samples. I am not sure what effect this has.
Bootstrap Method:Time Series of N Discrete Events
![Page 12: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/12.jpg)
![Page 13: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/13.jpg)
![Page 14: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/14.jpg)
![Page 15: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/15.jpg)
![Page 16: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/16.jpg)
![Page 17: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/17.jpg)
![Page 18: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/18.jpg)
![Page 19: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/19.jpg)
![Page 20: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/20.jpg)
![Page 21: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/21.jpg)
Piecewise Constant Model(partitions the data space)
Signal modeled as constant over each partition element (block).
![Page 22: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/22.jpg)
Optimum Partitions in Higher Dimensions
● Blocks are collections of Voronoi cells (1D,2D,...)● Relax condition that blocks be connected● Cell location now irrelevant● Order cells by volumeTheorem: Optimum partition consists of blocks
that are connected in this ordering● Now can use the 1D algorithm, O(N2)● Postprocessing: identify connected block fragments
![Page 23: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/23.jpg)
![Page 24: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/24.jpg)
BlocksBlock: a set of data cells
Two cases:● Connected (can't break into distinct parts)● Not constrained to be connected
Model = set of blocks
Fitness function:
F( Model ) = sum over blocks F( Block )
![Page 25: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/25.jpg)
Connected vs. Arbitrary Blocks
![Page 26: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/26.jpg)
![Page 27: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/27.jpg)
![Page 28: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/28.jpg)
2D Synthetic Bootstrap Example: Raw Data
![Page 29: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/29.jpg)
![Page 30: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/30.jpg)
![Page 31: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/31.jpg)
![Page 32: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/32.jpg)
Local Mean & Variance of Area/Energy (idea due to Bill Atwood)
![Page 33: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/33.jpg)
![Page 34: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/34.jpg)
![Page 35: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/35.jpg)
![Page 36: Analysis of Astrophysics and Particle Physics Data using ...scipp.ucsc.edu/seminars/experimental/Scargle.pdf · Analysis of Astrophysics and Particle Physics Data using Optimal Segmentation](https://reader033.vdocuments.net/reader033/viewer/2022041523/5e2f95f87aab3157b21fe27f/html5/thumbnails/36.jpg)