shakealert cisn testing center (ctc) development
DESCRIPTION
ShakeAlert CISN Testing Center (CTC) Development. Philip Maechling Information Technology Architect Southern California Earthquake Center (SCEC) 14 October 2010. CTC Progress in 2010. - PowerPoint PPT PresentationTRANSCRIPT
SCEC: An NSF + USGS Research Center
ShakeAlert CISN Testing Center (CTC) Development
Philip MaechlingInformation Technology ArchitectSouthern California Earthquake Center (SCEC)14 October 2010
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-present).
2. Changed our automated software testing infrastructure from web-based (Joomla) system to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
2
CTC Progress in 2010
EEW Testing Center Provides On-going Performance Evaluation
• Performance summaries available through login (www.scec.org/eew)• Evaluation results for 2010 include 144 M4+ earthquakes in CA Testing
Region• Cumulative raw summaries (2008-present) posted at
(scec.usc.edu/scecpedia/Earthquake_Early_Warning)
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-present).
2. Changed our automated software testing infrastructure from web-based (Joomla) system to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
4
CTC Progress in 2010
Earthquake Catalog
Retrieve Data
FilterCatalog
Filtered Earthquake
Catalog
CISN Testing Center (CTC) Forecast Evaluation Processing System
CISN EEW Testing Center (CTC) and Web Site
ANSS Earthquake
Catalog
CISN Decision Modules
CISN User Modules
Load Reports
ShakeAlert Earthquake ParameterForecast
ShakeAlert Ground Motion
Forecast
Observed ANSS EQ Parameter and
Ground Motion Data
ShakeAlert Forecast
Information
Evaluation testscomparing
Forecasts andObservations
ShakeMap RSS Feed Ground Motion
Observations
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-present).
2. Changed our automated software testing infrastructure from web-based (Joomla) to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
8
CTC Progress in 2010
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-present).
2. Changed our automated software testing infrastructure from web-based (Joomla) to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
10
CTC Progress in 2010
11
Current ShakeAlert CTC retrieves ShakeMap RSS data and plots observations for all Mag 3.0+ earthquakes in California Testing Region as shown (left).
CTC Progress in 2010
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-Present)
2. Changed our automated software testing infrastructure from web-based (Joomla) to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
12
CTC Progress in 2010
Earthquake Catalog
Retrieve Data
FilterCatalog
Filtered Earthquake
Catalog
CISN Testing Center (CTC) Forecast Evaluation Processing System
CISN EEW Testing Center (CTC) and Web Site
ANSS Earthquake
Catalog
CISN Decision Modules
CISN User Modules
Load Reports
ShakeAlert Earthquake ParameterForecast
ShakeAlert Ground Motion
Forecast
Observed ANSS EQ Parameter and
Ground Motion Data
ShakeAlert Forecast
Information
Evaluation testscomparing
Forecasts andObservations
ShakeMap RSS Feed Ground Motion
Observations
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-Present)
2. Changed our automated software testing infrastructure from web-based (Joomla) to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
14
CTC Progress in 2010
15
CTC Progress in 2010
Initial CTC Evaluation Test is defined in 2008 CISN EEW Testing Document (as updated July 2010). Previous Algorithm Testing Center did not implement this summary. Access to ShakeMap RSS ground motion observations makes automated implementation practical.
1. Prioritization of forecast evaluation tests to be implemented2. SCEC science planning of EEW forecast evaluation experiments3. Use of EEW in time-dependent PSHA information4. Consider Extending ShakeMap format as CAP-based forecast
exchange format.– Send forecasts information (and time of report) to produce:– ShakeMap Intensity Maps– ShakeMap Uncertainties Maps
5. Consider ShakeAlert interfaces to support comparative EEW performance tests. Provide access to information for each trigger:
– Stations Used In Trigger– Stations Available when declaring Trigger– Software Version declaring Trigger
16
Scientific and Technical Coordination Issues
End
17
Proposed CTC Evaluation Tests
18
Rigorous CISN EEW testing will involve the following definitions:– Define a forecast– Define testing area– Define input data used in forecasts– Define reference observation data– Define measures of success for forecasts
19
Design of an Experiment
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
20
Proposed Performance Measures
Summary 1.1: Magnitude X-Y Diagram
Measure of Goodness: Data points fall on diagonal line
Relevant: T2,T3,T4Drawbacks: Timeliness
element not representedWhich in series of magnitude
estimates should be used in plot.
21
Experiment Design
Summary 1.2: Initial magnitude error by magnitude
Measure of Goodness: Data points fall on horizontal line
Relevant: T2,T3,T4Drawbacks: Timeliness
element not represented
22
Experiment Design
Summary 1.3: Magnitude accuracy by update
Measure of Goodness: Data points fall on horizontal line
Relevant: T3,T4Drawbacks: Timeliness
element not represented
23
Experiment Design
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
24
Proposed Performance Measures
25
Experiment Design
Summary 2.1: Cumulative Location Errors
Measure of Goodness: Data points fall on vertical zero line
Relevant: T3, T4Drawbacks: Does not
consider magnitude accuracy or timeliness
Summary 2.2: Magnitude and Location error by time after origin
Measure of Goodness: Data points fall on horizontal zero line
Relevant: T3, T4Drawbacks: Event-specific
not cumulative
26
Experiment Design
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
27
Proposed Performance Measures
28
Experiment DesignSummary 3.1 : Intensity Map
ComparisonsMeasure of Goodness: Forecast
map matches observed mapRelevant: T4Drawbacks: Not a quantitative
results
Summary 3.2: Intensity X-Y Diagram
Measure of Goodness: Data points fall on diagonal line
Relevant: T1,T2,T4Drawbacks: Timeliness
element not representedWhich in series of intensity
estimate should be used in plots T3.
29
Experiment Design
Summary 3.3: Intensity Ratio by Magnitude
Measure of Goodness: Data points fall on horizontal line
Relevant: T1,T2,T4Drawbacks: Timeliness element
not representedWhich intensity estimate in
series should be used in plot.
30
Experiment Design
Summary 3.3: Predicted to Observed Intensity Ratio by Distance and Magnitude
Measure of Goodness: Data points fall on horizontal line
Relevant: T1,T2,T4Drawbacks: Timeliness element
not representedWhich intensity estimate in
series should be used in plot.
31
Summary 3.3: Evaluate Conversion from PGV to Intensity
Group has proposed to evaluate algorithms by comparing intensities and they provide a formula for conversion to Intensity.
32
Summary 3.4: Evaluate Conversion from PGV to Intensity
Group has proposed to evaluate algorithms by comparing intensities and they provide a formula for conversion to Intensity.
33
34
Experiment DesignSummary 3.5: Statistical Error
Distribution for Magnitude and Intensity
Measure of Goodness: No missed events or false alarms in testing area
Relevant: T4Drawbacks:
35
Experiment DesignSummary 3.6: Mean-time to
first location or intensity estimate (small blue plot)
Measure of Goodness: Peak of measures at zero
Relevant: T1,T2,T3,T4Drawbacks: Cumulative and
does not involve accuracy of estimates
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
36
Proposed Performance Measures
37
Experiment DesignNo examples for System Performance Summary defined as
Summary 4.1: Ratio of reporting versus non-reporting stations:
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
38
Proposed Performance Measures
39
Experiment DesignSummary 5.1: Missed event and
False Alarm MapMeasure of Goodness: No
missed events or false alarms in testing area
Relevant: T3, T4Drawbacks: Must develop
definitions for missed events and false alarms, Does not reflect timeliness
40
Experiment DesignSummary 5.2: Missed event and
False Alarm MapMeasure of Goodness: No
missed events or false alarms in testing area
Relevant: T3, T4Drawbacks: Must develop
definitions for missed events and false alarms, Does not reflect timeliness
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
41
Proposed Performance Measures
42
Experiment DesignSummary 6.1: Missed
Event mapMeasure of Goodness: No
missed events in testing region
Relevant: T3, T4Drawbacks: Must define
missed event. Does not indicate timeliness
End
43