climate data records cdrs and cics j.j. bates, j.l. privette, e.j. kearns, noaa national climatic...
TRANSCRIPT
Climate Data Records
CDRs and CICS
J.J. Bates, J.L. Privette, E.J. Kearns, NOAA
National Climatic Data Center
Asheville, NC
Outline
• NOAA Climate Goal reaction to NRC reports• Summary and status of NOAA’s CDR Program• Impact of CICS on CDR Program• Progress in JPSS-specific CDR issues and
strategic planning• F11 FFO
2
A Climate Data Record (CDR) is a time series of sufficient length, consistency, and continuity to determine climate variability and change FCDR: Calibrated signals for a family of sensors together with
the ancillary data used to calibrate them. (e.g. brightness temperatures, TOA radiances)
TCDR: Geophysical variables derived from FCDRs; May be generated by blending satellite observations, in-situ data, and model output (e.g., Aerosol Optical Depth, SST)
A Climate Information Record (CIR) is derived from CDRs and related data tand provides specific information about an environmental phenomena of importance to science and society (e.g., Hurricane tracks, Arctic Sea Ice Extent, Incidence of disease)
NOAA’s CDR Program is in response to an operational need
CDRs Provide Scientifically Corrected Long Time Series for Climate Analysis
Operational weather and hazard products are produced rapidly to potentially save life and property
Climate Data Records (CDRs)provide long term product consistency through rigorous reprocessing with advanced algorithms, ancillary data and evolved instrument understanding.
Climate Information Records (CIRs)provide specific information about environmental phenomena of particular importance to science and society (e.g., hurricane trends, drought patterns)
Ve
ge
tatio
n G
ree
nn
ess
In
de
x
Uncorrected Data Time Series Contain Both Environmental Information and
Satellite-induced Artifacts
New satellite
launched
Time (year)
Top of Atmosphere Vegetation Index
Top of Canopy Vegetation Index
Operational Climate vs
Operational Weather
5
CDR Program functionally located within RSAD at NCDC
NOAA NESDIS & Climate Goal reaction to NRC “Ensuring the Climate Record…”
• Clear Agency Roles and Responsibilities– Regular quarterly NOAA-NASA meetings and
NOAA participation in CERES and SORCE science team meetings are helping
• International Coordination– NOAA provides international leadership through
WMO GSICS & SCOPE-CM as well as CEOS Climate ad hoc Group
• Community Involvement in the Development of Climate Data Record
6
Management Board USGCRP Observations Management Structure
Product Development Teams (CDRs, CIRs and Support)
Engaging User Community
Modeling Monitoring Prediction Research
Experts in Instrument Characterization, Algorithms, Validation, Data
Management, Applications, and Observing System Performance Monitoring
Oth
er A
genc
ies
GE
OS
S, C
EO
SIn
tl D
ata
Pro
gram
s
Lev
erag
ing
Res
ou
rces
/ C
oll
abo
rati
on
Observing System
Performance Monitoring
Production of Near Real-time
CDRs
Processing of CDRs for Long-term Records
Climate Information
Records
CDR Stewardship
Improved CDR and
CIRs
Design for Future
Systems
CDR Program Functional Framework
End-to-End Stewardship
8
END-TO-END STEWARDSHIP OF NOAA SATELLITE OBSERVATIONS
REQUIREMENT INPUT/OUTPUT QUALITY ASSURANCE ARCHIVAL OUTCOMES & IMPACTS
ESSENTIAL CLIMATE VARIABLE
OBSERVING SYSTEM OR
SENSOR
INSTRUMENT HEALTH
MONITORING
RAW DATA
RECORDS
ENVIRONMENTAL
DATA RECORDS
CALIBRATION &
VALIDATION
ARCHIVE STEWARD
SHIP
CLASS IT DEVELOP
MENT
CLASS OPERATIONS
CLIMATE DATA
RECORDS
CLIMATE SERVICES:
MONITORING UNDERSTANDING
MODLEING
Upper Air Temp, SST, Clouds, Sea
Ice… NPP / JPSSPARTY
RESPONSIBLE
ERB/TSI, Ozone
Climate Sensors (TSIS, CERES,
OMPS)
Cloud Properties GOES-R Sea Level JASON-3
Surface WindOSVW (Quickscat
replacement)
PrecipitationGCOM-W AMSR-
2 Ocean Color GCOM-C SGLI
Upper Air Temperature
COSMIC-2 / GPS-RO
NOTIONAL SAMPLE
CDR Activities Address 3 Epochs
1970 1980 1990 2000 2010 2020 2030
POES/GOES NPP JPSS/JASON-3/GOES-R
Establish operational CDRs and CIRs from NOAA’s archive of four decades of heritage data
Ensure climate quality data from operational systems, build Raw Climate Data Records to facilitate future
reprocessing, ensure continuity with heritage CDRs
Prepare for products from “Remanifested”
climate sensors and future missions
Long-Term Planning
• NOAA is initiating long-term planning by initiating community requirements meetings– Continuity of Earth Radiation Budget (CERB), July 2010
• NOAA is working with the international community to coordinate efforts– The aim of the WMO Sustained, Co-Ordinated Processing of
Environmental Satellite Data for Climate Monitoring (SCOPE-CM) is to establish a network of facilities ensuring continuous and sustained provision of high-quality satellite products related to the Essential Climate Variables (ECV), on a global scale, responding to the requirements of the Global Climate Observing system (GCOS).
– Committee on Earth Observing Satellites Climate Ad hoc Group (CEOS-CAG) - The group should ensure the more coherent product of FCDR/ECVs among agencies in response to the defined needs of GCOS
10
Preparing for CERES - ISCCP Reprocessing Data Flow
Multi-National Geostationary Satellite Data
Perform Mapping Operation
Run Cloud Detection Algorithm
Perform Radiative Retrievals
10 km pixel data
every 3 hours
Calculate Gridded Statistics 1 deg res
every 3 hours Merge the Various Satellites
Calculate Monthly Averages
B1
B4
BX
DX
DS
D1
D2 1 deg monthlyNCDC has
processed to this level to SW and LW Flux
calculations
CDR Program Funding Profile
Funding Mark or Source
Fiscal Year President’s Budget Request ($M)
Appropriated($M)
Redirected ($M)
ARRA($M)
2007 0 0 3.5 0
2008 0 0 0 0
2009 0 0 2.9 5.0
2010 7.0 10.0 0 0
2011 18.0 * * *
* To Be Determined
FY10 Investment Allocation*
13
Activity Investment (%)
Product Development 50
IT Systems 1
Research-to-Ops 5
Production and Stewardship
8
Science & CIRs 11
Mgmt./Planning/Outreach and Facility
26
100
*Expected**Anticipated
FY10 Organization Allocation*
14
Category Obligation (%)
Other NOAA (BOPs) 13
Other Federal Agencies 7
Universities: CICS-NC 23
Universities: Other CIs 14
Universities: Non-CI 11
Contracts 24
FTEs (NCDC) 6
100
13 7
231411
24 6Other NOAA (BOPs)Other Federal Agencies Universities: CICS-NC Universities: Other CIsUniversities: Non-CI Contracts FTEs (NCDC)
48%
* planned
NOAA NESDIS & Climate Goal reaction to NRC “Ensuring the Climate Record…”
• Microwave CDRs funded• Altimeter focus in FY11 FFO• Clouds and radiation (ISCCP)
reprocessing and R2O underway
• Hyperspectral being used to correct filtered radiometers
• Solar irradiance CDR funded• Global aerosol CDR R2O• Ocean color currently not
funded• Ozone CDR team funded
15
CDR Product Discipline Teams
• Goal– Have community experts coordinate activities within discipline/sensor– Develop coherent and self-consistent product suites– Ultimately seek consistency across disciplines (e.g., cloud detection)
• Possible Issues for Team to Address– Algorithms – adopt common cloud/land/water masks, FCDR and ancillary
data inputs, consistent QA flags and metadata for quality control, begin algorithm chain production and testing in development phase
– Products – identify gaps or resolve apparent redundancies within product suite; adopt common and appropriate grid projections, resolutions, data formats and packaging, etc.
16
Current CDR Development Portfolio(FY09 & FY10 Awards)
AVHRR (VIIRS)• Cloud Properties (Kato)• Snow/Ice (Key)• VNIR Cal./Clouds Minnis)• Thermal Calibration
(Mittaz)• Land/Carbon (Vermote)• Ocean Fluxes (Clayson)
SBUV (OMPS)• Ozone (Flynn)
DMSP: SSM/I, SSMIS (MIS)• Calibration (Kummerow)• Snow/Ice (Key)• Water Vapor (Luo)
SORCE, Glory (TSIS)• Solar Irrad.
(Pilewskie)
GOES: Imager (ABI)• VNIR Cal./Clouds (Minnis)
AMSU (ATMS)• Hydro Cycle (Ferraro)• Upper Air Temp (Ho)• Water Vapor (Luo) • Temp. Profile (Zou)
Other Satellites
Arrows identify key climate instruments
HIRS (CrIS)• FCDR/Intersensor calibration (Cao)• Water Vapor (Luo)• Cloud Properties (Menzel)
ERBS: ERBE (CERES)• Radiation Budget (Kato)
GPS RO (Various)• Temp. Profiles (Ho)
18
Coordination Activities:Annual CDR Team Meeting
• 2-day meeting (Aug. 4-5) in Asheville
• Attended by more than 50 participants from academia, industry, NOAA and other agencies
• Featured 14 competitively-selected CDR investigators describing algorithm and product plans
• CDR Program rolled out research-to-operations plans and expectations from team
• Impact: CDR investigators identified intra-team product dependencies & synergies; CDR Program developed detailed understanding of CDR algorithm formulation and maturity
19
TCDRsAtmospheric
Profiles Clouds & Aerosols PrecipitationOzone/Trace
Gases Oceans Land Cryosphere Solar Irradiance Radiation Budget
Discipline Lead Menzel Heidinger Ferraro Flynn Evans Vermote Key Pilewski Zhao
NCDC Tech. Lead Shi Knapp Nelson Shi Banzon Guillevic Zhang Zhao Zhao
NCDC IT Lead TBD TBD TBD TBD TBD TBD TBD TBD Hankins
Menzel (Shi)Heidinger (Knapp)
Boyles (Nelson) Flynn (Shi) Evans (Banzon)
Vermote (Guillevic) Key (Zhang) Pilewski (Zhao) Rossow (Zhao)
Ferraro (Kim) Minnis (Knapp) Ferraro (Kim) Long (Shi) Minnet (Zhang) Rossow (Zhao)Robinson (Guillevic) Lean (Zhao) Kato (Zhao)
Ho (Shi)Sarooshian
(Kim) Rosenlof (Shi)Reynolds (Banzon) Meier (Zhang) Pap (Zhao) Lee (Shi)
Luo
(Knapp/Young) Casey (Banzon) Lazlo (Zhao)
Zou (Shi) Clayson (Zhang) Li (Zhao)
Mears (Nelson) Heidinger (Zhao)
FCDRs Imagers (SW/IR)Imagers (u-
wave) Sounders (TIR)Sounders (u-
wave)Sounders (Trace
Gas)Solar
Radiometers ERB Instruments GPS-RO
Team Leader Evans Kummerow Cao Ferraro Flynn Pilewski Kato Ho
NCDC Tech. Lead Knapp Semenegus Shi Kim Shi Zhao Zhao Shi
NCDC IT Lead TBD TBD TBD TBD TBD TBD TBD TBD
Evans (Banzon)Kummerow
(Semenegus) Cao (Shi) Ferraro (Kim) Flynn (Shi) Pilewski (Zhao) Kato (Zhao) Ho (Shi)
Mittaz (Knapp)Wentz
(Semenegus) Menzel (Shi) Luo (Nelson) Rice (Zhao) Rossow (Zhao)
Vermote
(Guillevic) Lee (Shi) Zou (Shi) Lean (Zhao)
Minnis (Knapp)
Heidinger (Knapp)
New CDR Teaming at NCDC
Public Posting of Prototype CDRs in Operations
20
• Maturity Matrix adopted by GCOS (1 of 12 guidelines for CDR Production)
CDRs in Development
21
Online Data• Investigator Teams/
Affiliations• Abstracts• Progress Reports• Community Workshop URLs• Team Meeting Presentations
Impacts• Used by EUMETSAT’s
Satellite Application Facility on Climate Monitoring (CM-SAF)• Interest in coordinating work
CDR Program Management: Staffing
• Federal Program Manager vacancy announcement open– FAC-P/PM Senior certification– Closes 9/10
• RFP for Program Management Support released by ERAD– 2 year base + 1 year option– 4 Key Personnel positions– Technical review completed Aug. 20– Expected award date: this week?
22
Interaction with CICS
• Several selected CDR grants were awarded as Type III funding
• Support for key internal work– NEXRAD, extreme events, SST
• Securing science and scientific
programmer support in RSAD• Postdocs and students• Assembling IT Development Environment• Summer ‘11 climate workshop
Impact of Interaction with CICS
• Introduction of academic flavor to an operational federal workplace
• Part of paradigm shift at NCDC• Enabling CDR processing
– Development environment• Network connections• Bring another dimension to NCS
JPSS-Specific Issues
• NASA missions are end-to-end and NOAA are broken into segments– NOAA has mapped one to the other and
found gaps that are being addressed• Sensor integration and testing• Instrument scientist and analyst• Sensor cal/val teams• CDR legacy processing not portable
25
CDR Information Flow
Sensor DataRecords (SDRs)
Data (Direct & Remotely Sensed- RDRs)
Fundamental Climate Data
Records (FCDRs)
Thematic Climate Data Records (TCDRs)
Homogenization and Calibration
Time-tagged Geo-Referenced
EnvironmentalData Records
(EDRs)
Converted to Bio-Geophysical
Variables
Climate Information Records (CIRs)
Climate Data Records or Homogenized Time Series
Fundamental Climate Data Record (FCDR): Time series of calibrated signals for a family of sensors together with the ancillary data used to calibrate them.
Thematic Climate Data Record (TCDR): Geophysical variables derived from FCDRs, often generated by blending satellite observations, in-situ data, and model output.
Raw Climate Data Records (R-CDRs)
Recent AccomplishmentsMajor Milestones
Project Schedule
• System Definition Review (Completed: Mar. 2010)
• Critical Design Review (Scheduled: Sept. 16, 2010)• “R-CDR CDR” • External reviewers from GSFC & Univs. Miami,
Wisconsin & Colorado
Updated: 03/12/10
Project Risks
• Complexity of the NPP/NPOESS RDRs and short schedule
• Staffing needs for ATMS, and OMPS NP Raw CDRs and system execution coding
• Hired 2 new programmers; 1 more sought
• Successfully ported and adapting NPP/VIIRS and CrIS software as used by NASA PEATES
R-CDR
Interface Data
Processing Segment
Land PEATE
NPP Operational Software Reuse
Climate Raw Data Record(R-CDR) Status
NPP/JPSS GSFC NCDC
FY11 FFO
• http://www.federalregister.gov/a/2010-17294/p-579 • Initially focused on Fundamental CDRs and Thematic CDRs related to
Earth's water and energy cycles and sea level • Funding Availability: For the present grants competition, the CDR Program
expects to select proposals over a two year period (FY 2011 and FY 2012) for funding. The total anticipated Federal funding in FY 2011 is $2.5M for new awards. The anticipated number of new awards is from 5 to 15.
• The CDR Program anticipates new funding availability in FY 2012 for additional awards from the present grants competition.
• The total anticipated Federal funding in FY 2012 and the number of additional awards will be dependent on the enacted budget.
• Application Deadline: Letters of Intent (LOI) should be received at the CDRP Office no later than 5 p.m. Eastern Time, September 15, 2010
• Full proposals must be received no later than 5 p.m. Eastern Time, November 10, 2010
Summary
• The CDR Program has recurring resources and is actively engaging the U.S. and international climate communities to operationalize CDRs for NOAA.
• CICS has already proven to be a key cooperator for the CDR Program
• JPSS gaps in climate information stewardship have been identified and are being addressed
• FY11 FFO is out -- and responses due soon!
29
Questions?
NPP Data Flow Diagram
NESDISCentral
AFWACentral
NPP/NPOESS Data ProcessingSegment (IDPS)
NPP/NPOESS Data ProcessingSegment (IDPS)
Archive & Dist Segment
(CLASS; NCDC and NGDC)
Archive & Dist Segment
(CLASS; NCDC and NGDC)
Ingest
RoboticStorageDatastore
Inventory
Science Data
Segment(NASA)
Science Data
Segment(NASA)
Cal/Val ProgramCal/Val
Program
ERB CARS
NDE(NPOESS
Data Exploitation)
NDE(NPOESS
Data Exploitation)
PEATEs
Cal.
Production &
Validation
STAR/GSICS
DataProcessing
&Re-Processing
Char.
CDR
CIR
RDRs
xDRs, IPs, Ancillary& Auxiliary Data
SDRs, EDRs, IPs,
Ancillary &
Auxiliary Data
UsersUsers
CERES CDRs
NUPs
KEY:RDR Raw Data RecordSDR Sensor Data RecordTDR Temperature Data Record EDR Environmental Data RecordxDR RDR, SDR, TDR, EDRCDR Climate Data RecordCIR Climate Information RecordIP Intermediate ProductNUP NOAA Unique Product
xDRs, IPs
V1.6 NCDC NPP 16 July 2009
RDRs
CDRs,CIRs
Research Communit
y
Research Communit
yOMPS-Limb SDRs, EDRs
SDRs, TDRs,
IPs, EDRs
xDRs, CDRs, NUPs
CERES CDRs
PEATEsPEATEsPEATEs
Correlative
Data &
Results
CDR Program (NCDC)CDR Program (NCDC)
CIRs
, IPs
, Anc
illar
y&
Aux
iliar
y D
ata
xDRs, IPs, Ancillary& Auxiliary Data
RDRs, Ancillary& Auxiliary Data
R-CDRs
Climate Raw Data Record (C-RDR)System Interfaces
xDRs, IPs, Ancillary& Auxiliary Data
CDR/CIR Processing
CLASS NCDC R-CDR
Processing
Archive
National Climatic Data Center (NCDC)
NESDIS Central
Interface Data
Processing
Segment (IDPS) CLASS PoP
Climate User Community
Outline
• About NOAA’s new CDR Program at the National Climatic Data Center
• Why NOAA is interested in this activity for oceans, land, & atmosphere
• What is meant by an operational CDR• How it will engage the science
community to meet program goals
FY10 CDR Investments
34
Area DescriptionPrimary
Vehicle(s)
Development – Products Algorithms, Prototype Products, Docs
Grants, CI, IAAs
Development – IT System Computing, Networks Contracts
Research-to-Operations Code conversion, optimization, IV&V
CI, Contracts
Production & Stewardship
(Re-)Processing O&M, Archive, Distribution
Contracts
Science and CIRs CDR analysis and characterization CI
Mgmt/Planning/Outreach and Facility
Incl. Project docs, Summer Institute, User Reqts. Group, Renovation
Contracts, CI
Maturity Matrix Identifies Milestones and Research-to-Operations Transition Points
Sustained, Coordinated Processing of Environmental Satellite Data for Climate Monitoring (SCOPE-CM)
The aim of SCOPE-CM is to • establish a network of facilities ensuring continuous and sustained provision
of high-quality satellite products related to the Essential Climate Variables (ECVs)
• global scale• respond to the requirements of the Global Climate Observing system
(GCOS)