comprehensive large array-data stewardship system status presented to daarwg kern witcher class...

Comprehensive Large Array-data Stewardship System

Status

Presented to DAARWG

Kern WitcherCLASS Program Manager

November 8, 2012

Agenda• CLASS Summary• Program Overview• Data Center Migration• CLASS Capabilities• CLASS System Evolution• Other Initiatives• Challenges• CLASS and DAARWG

2

CLASS Summary• CLASS meets original intended purpose (Large Volume Data)

• Suomi-NPP• Volume Ingested – 1.32Pb• Volume delivered-3.53Pb• Files Delivered- over 34M

• 22 major datasets included• AVHRR, GOES, IASI, CFS-R

• 2.27 Pb, safely stored at 2 sites

• CLASS must evolve to become a sustainable NOAA enterprise • NOAA’s desire for CLASS to be the Enterprise Archive• Significant challenges must be addressed

3

10/1111/1112/11 1/12 2/12 3/12 4/12 5/12 6/12 7/12 8/120

100

200

300

400

500

600

S-NPP Archive and Subscription De-livery

Archive (single node)Subscriptions (point to point & internet)

Month

Volu

me

(Ter

abyt

es)

Program Overview

4

Program Transition • Program/ Contract Management was transitioned from OSD to NCDC in May of 2011• New WBS and program management staff established• Increased communications between CLASS development and Data Centers• CY3* extension in Fall of 2011 to end of FY12• CY4 extension though Q1 of FY13• CY5 to begin Q2 of FY13. CY5 delayed to allow for

• CLASS response to emerging Data Center consolidation• data migration activities• begin “pivot” or evolution of CLASS to the Enterprise Archive System• Effort is dependent on final funding allocations for FY13

• Contract may reach ceiling as early as FY15• Acquisition planning needs to begin by January 2013

5

*- Contract Year

Organization

NGDC NODC

CLASS Operations Planning

Board

COWG

NCDC

Other NESDIS

Contractor

Legend

Supervisory

Reporting

Remote Sensing Applications Division

CLASS Program Manager – 100%(Kern Witcher)

NCDC

IT Specialist/Systems Engineer - 100%(Jay Morris )

Support ServicesDivision

Budget Analyst– 5%(L. Cholid)

Inventory Spec – NCDC – 10%(A. Annis, acting )

OMB Exhibit 300 – 30%(T. Cohen)

Budget Execution - 5%(T. Leary)

COR / Proj Engineer + CWIP – 100%(Jim Goudouros )

ISSO – 100%(Scott Koger)

Alt COR – 5%(J. Niemiec )

Government Task Monitor – 30%(N. Ritchey)

Operations Manager – 100%(D. Carter))

NESDISCIO

Total of 5 dedicated personnel6

Current Program Milestones

Notes:CDR: Critical Design ReviewFOR: Flight Operations ReviewPDR: Preliminary Design ReviewSDR: Systems Definition Review SRR: System Requirements ReviewORR: Operational Readiness ReviewNCT: NPP Connectivity TestICD: Interface Control Document

Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

CLASS

GOES-R Campaign

NPP Campaign

Climate Model Data

Nexrad Campaign

NDE Campaign

Jason-3 Campaign

Data Center Migration

CLASS Releases

FY15 FY16 FY17Fiscal Year FY09 FY10 FY11 FY12 FY13 FY14

Development Construction Integration/ Test Operations

ORRArchive &

Access PDRSRR

ORR

J-1 CDR

J-1 SRR J-1 PDR ORR

6/15

CDR v1 CDR v2

PDR

Charter ORR

ICD

CDR

CDR

10/11

PDRSDR ORR

2/12 9/126/12

SRR

7/11

Launch

10/16

Launch

4/13

Launch

10/15

4/14 10/14 4/15 4/16

R5.2

10/15

R5.3 R5.4 R5.5

10/16

Current MilestoneCompleted Milestone

SDS v1

R3 R4

Interface PDRArchive &

Access CDR

Interface CDR

LaunchFORNCT3

NCT4

NEAAT 1

JPSS-1

ICD

R6.0

NPP Launch

NEAAT 2

NEAAT 3

NEAAT 4

PDR CDR

IDPS Blk 1.5 CDR

IDPS Blk 2.0 CDR

IDPS Blk 1.5 Ops

IDPS Blk 2.0 Ops

7

Currently focused on NESDIS and NWS data sets

Ongoing Challenges

• Architectural Boundary Responsibilities• Common vs. Unique Data Ingest• Data Center Migration• Data Center Consolidation• Governance• Budget• Transition to New Architecture

8

CLASS Definition

Purpose: Support long-term, secure storage of NOAA-approved data, information, and metadata and enable access to these holdings through both human and machine-to-machine interfaces. Capabilities will be provided in 3 primary functional areas (Open Archive Information System Reference Model, OAIS-RM components):

–Ingest - provide mechanisms by which data, information, and metadata are transferred to and organized within the storage system.

Issue:• Feasibility – Cannot afford/maintain unique solutions

9

From CLASS Level 1 Requirements (preliminary) signed 2008

CLASS Definition (con’t)Archival Storage - provide common enterprise means for data, information, and metadata to be stored by the system and the capability to refresh, migrate, transform, update, and otherwise manage these holdings as part of the preservation process.

Access - provide common enterprise access capability enabling users to identify, find, and retrieve the data and information of particular interest to the user.

Issue: • Open to interpretation – IT systems versus Data Center

stewardship responsibilities

• Optimal data access varies by observations (vertical profiles vs. spatial patterns, Gridded vs. swaths, Time series vs. synoptic)

10

11

Goals: As an enterprise solution, CLASS will reduce anticipated cost growth associated with storing environmental datasets by:– Providing common services for acquisition, security, and

project management for the IT system supporting NOAA Archives

– Consolidating stove-pipe, legacy archival storage* systems – Relieving data owners of archival storage-related system

development and operations issues

Archival storage provides the services and functions for the storage, maintenance and retrieval of archival information packets. Archival storage functions include receiving archival information packets from ingest and adding them to permanent storage, managing the storage hierarchy, refreshing the media in which archive holdings are stored, performing routine and special error checking, providing disaster recovery capabilities, and providing archival information packets to access to fulfill orders

11

CLASS Goals

High Level Requirements for CLASS

5.1 Core Mission Requirements 5.1.1 CLASS shall provide defined and documented human and machine-to-machine interlaces by which archives may securely store, maintain, and provide access to their data, information, and metadata holdings for indefinite periods. 5.1.2 CLASS shall ingest, provide long-term, secure storage, and provide access to baseline information holdings 5.1.3 CLASS shall provide long-term, secure storage of and common access to information pertaining to processing of CLASS maintained information holdings, including documentation, processing algorithms, and procedures. 5.1.4 CLASS shall comply with applicable National Archives and Records Administration (NARA) regulations. 5.1.5 CLASS shall initiate pilot programs with the GEO-IDE project to support risk reducing development and phased integration of standards for metadata, machine-to-machine interlaces, and archives.

From CLASS Level 1 Requirements (Preliminary 2008)

12

Proposed NOAA Action: Finalize CLASS Level 1 Requirements

Current CLASS Governance

• CLASS is designated a "major" NOAA system in accordance with criteria in NOAA's Administrative Order (NAO) 216-108.

• NOAA Observing Systems Council (NOSC) is the designated oversight council and will review the CLASS project at each Key Decision Point (KDP).

• The project will be reviewed as a major project by the Program Management Council (PMC), unless delegated.

• As an IT project, the CLASS project will be reviewed by the NOAA Information Technology Review Board (NITRB) and by the Commerce Information Technology Review Board (CITRB).

• Changes to project baseline, scope, and direction shall be approved by the Deputy Undersecretary for Oceans and Atmosphere

From the CLASS Level I Requirements

13

The CLASS Program has numerous oversight and approval authorities

Proposed NOAA Action: Provide Clear Lines of Authority and a well defined Governance Structure


14

Goal, Objective, and Outcomesfrom NOAA Data Centers Data Migration Plan v1.2 signed May 2011

• Goal: Prior to the end of FY 2015, the CLASS Operational System will be the primary safe storage/access capability for all environmental data holdings under the auspices of NOAA’s Data Centers.

• Objectives: The objective of this plan is migration of all historical environmental data holdings on current Data Center storage systems to CLASS, in addition to the ingest into CLASS [of] near real-time environmental data streams currently received by the Data Centers. This objective involves archival storage as well as elements of ingest, data access, data management, and preservation planning.

Data Center Migration Status

15

Starting point

• Incoming Data Streams already migrated to CLASS include– POES, GOES, Jason-2, MetOp

• Incoming Data Streams solely in CLASS– Suomi-NPP, GCOM-W, (future GOES-R, future JPSS, future Jason-3)

Data Center Migration Status(con’t)

16

Data Center Status• Each Data Center developing its own individual

plans including schedule, metrics and milestones• CLASS PM included several pilot activities in

CLASS’ Contract Year 4 (CY4)– Cloud access pilot (NODC, NCDC) - ongoing– NEXRAD pilot (NCDC) - completed– NCDC in situ migration prototype activity - completed

• Next steps to be included in CLASS’ CY5 activities


17

Data Center Migration Status(con’t)CLASS accomplishments in FY12 for Data Center Migration

• Aerospace completed Phase I (“as is” analysis) of Data Center Con Ops

• Completed Data Center Migration Requirements Documents

• Completed Data Center Migration Interface Control Documents

• Migrated NCDC Historical Data Set• Completed NexRad Pilot 2 week migration test• Initiated Cloud Pilot Project

18

• Uncertainty regarding FY15 timeline– Reductions in CLASS funding available for migration

activity could delay implementation• Increasing comms costs, operational costs• FY13 President’s Budget has increased ORF to solve this

– Data Center consolidation activities– Individual Data Center plans not yet finalized

• Several possible technological solutions still being evaluated with Data Centers and CLASS


19

Summary• The Data Centers have an overall plan in place for the migration of

their historical environmental data to CLASS by FY15.• Many historical data sets already migrated – 60% by volume – but

many smaller sets still need to be completed. • Several large data streams already migrated, but many incoming

data streams still need to be transitioned to CLASS storage.• Each Data Center working on their individual plans and coordinating

with CLASS PM.• CLASS PM will address next steps in FY13/CY5, contingent upon

available funds for migration activities.


20

Current Capabilities

21

Current CLASS Assets

CLASS NGDC, Boulder COCLASS NSOF, Suitland MDCLASS NCDC, Asheville NC

Replication via NOAA Science Network(N-WAVE)

Users Users

Development TeamFairmont, WV

Functions

Operations• Ingest• Storage (Disk & Tape)• Public Access

Test and Integration Environment

Satellite Landing Zone

Development Environment

Direct Connectivity to:• ESPC- NOAA Environmental Satellite Processing

Center• National Ice Center• NOAA Coast Watch • JPSS Interface Data Processing Segment (IDPS)

Key Capabilities:• Tape Library Capacity – 2- 10,000 tape robotic libraries with a

total storage capacity of 15.5 Pb (LT04 Tapes/ Native)• Spinning Disk Capacity- 2.1 Pb (NSOF, NCDC, NGDC)• 10 Gb/sec Internal Network Backbone• Redhat Linux OS Server Count- 48 (576 processors)• 10 Gb/Sec WAN (N-Wave)

22

CLASS interfaces to NPP and JPSS-1

OPS RN

I&T RN

OPS RN

I&T RN

NWAVE

OPS FSN

Boulder

OPS FSN

Asheville

= Receipt Node – Data delivery to archive= Full Service Node (Ingest), Archive, and User Access

Primary JPSS IDPS

Backup IDPS at CBU

NSOF Fairmont

= CLASS backbone WAN

GRAVITE IFCs

Backup system for COOP Event

FUTUREIDPS Block 1.5/2.0

NPP, J-1, J-2CLASSNGDC

CLASSNCDC

STAR

CLASSNSOF

SDS

IDPS

GRAVITE

NWAVE 10Gb/sC3S

CURRENTIDPS Block 1.2

NPP

23

CLASS Interface to GOES-RGOES-R

(In Development)

24

Data Access from CLASS

Search and find info

Submit order

Submit service request

Submit ad-hoc order

Submit standing order

Consumer

25

Preservation Planning

Administration

Ingest Access

Data Management

ArchivalStorage

PRODUCER

CONSUMER

requests

results

Present CLASS

NPP Data Dissemination

CLASSNGDC

CLASSNCDC

STAR

CLASSNSOF

SDS

IDPS

GRAVITE

CLASS to SDSNear-instantaneous subscription delivery

(~3.3 TB/day)

NWAVE 10Gb/s

PublicSubscriptions

PublicAd hoc orders

IDPS to CLASS6-hour delay

C3S

Svalbard, Norway

~4.7TB/day

~4.7TB/day

CLASS to STARNear-instantaneous subscription

delivery(~3.3 TB/day)

CLASS to GRAVITENear-instantaneous subscription delivery

(~3.3 TB/day)

Replication~20-min

6-hour Delay imposed by CLASS1) Minimizes retransmissions requests2) Ensures control of limited distribution files.

Public SubscriptionsNear-instantaneous delivery

Public Ad Hoc Block Orders (up to 18 Tb/day)50% complete within 0 to 6 hours27% complete within 6 to 12hours22% complete within 12 to 72 hours 1% complete over 72 hours

Public Ad Hoc Bulk Orders Up to 7 days

Process Duration

IDPS to CLASS ~ 6 hours

File Ingest & catalog populated ~ 20 min

NCDC/NGDC Node Replication ~ 22 min

Total ~ 6:42

Note: Near- Instantaneous subscription delivery is limited by network bandwidth

26

Data Currently Archived in CLASS

27

Dataset Name Period of Record

Date First Archived into

CLASSTotal Volume

(TB) SourceSuomi National Polar-orbiting Partnership (S-NPP) 10/2011 to present 10/1/2011 1352.18 NASA/NOAAGeostationary Operational Environmental Satellite (GOES) 1/1979 to present 12/5/2003 310.00 NOAAClimate Forecast System - Reanalysis (CFS-R) 1/1979 to 3/2011 1/20/2010 294.56 NOAAAdvanced Very High Resolution Radiometer (AVHRR) 10/1978 to present 11/1/2001 99.49 NOAAInfrared Atmospheric Sounding Interferometer (IASI) 4/2007 to present 5/1/2007 93.84 EUMETSATDefense Meteorological Satellite Program (DMSP) 7/1987 to 2/1997 11/1/2001 26.53 DoDAdvanced Clear-Sky Products over Oceans (ACSPO) 5/2008 to present 5/1/2011 19.28 NOAACoast Watch 2/1989 to present 11/15/2005 15.70 NOAAAdvanced Scatterometer Level 1B (ASCAT) 2/2007 to present 2/1/2007 10.29 NASASynthetic Aperture Radar (SAR) 6/1992 to 3/2011 4/30/2003 9.03 NASA/NOAAContinuously Operating Reference Stations (CORS) 11/2010 to present 10/7/2011 7.79 NOAAJason-2 6/2008 to present 6/22/2008 6.68 NASAMicrowave Integrated Retrieval System (MIRS) 8/2007 to present 8/30/2007 6.45 NOAAOcean Color Data Products 5/2000 to present 5/7/2003 6.18 NOAATiros Operational Vertical Sounder (TOVS) 1/1988 to 12/1995 11/1/2001 5.74 NOAAGlobal Nav Satellite Receiver for Atmospheric Sounding Level 1B (GRAS) 9/2007 to present 9/14/2007 2.93 EUMETSATMicrowave Surface and Precipitation Products System (MSPPS) 7/2003 to present 7/20/2003 2.09 NOAAAerosol Optical Thickness (100 KM) (AERO100) 11/1998 to present 11/15/2005 0.98 NOAASolar Backscatter Ultraviolet Spectral Radiometer Version 2 3/1985 to present 5/23/2006 0.25 NOAAGlobal Change Observation Mission 1 - Water (GCOM-W1) 8/2012 to present 8/22/2012 0.04 JAXACoral Bleaching Monitoring Products (CORBL) 6/2007 to present 6/6/2007 0.00 NOAANCEP Weather Analyses and Forecast Charts 3/1999 to 5/2012 5/2/2012 0.00 NOAA

Total 2270.03

Represent 95% of the Archival Holdings by Volume

(Single Node)

Customer Groupings that Access CLASS

Jan Feb Mar Apr May Jun Jul Aug0

5

10

15

20

25

30

35

40

45

50

CommercialEducationalNOAANASADoDInternational

CLASS Users by Domain

TB

201228

Current Constraints

29

• Access limited through CLASS Web site• New data sets to archive require significant SW development

efforts• Implementation of Trusted Internet Connection (TIC) will effect

throughput and system performance• Access to data holdings have limited on-line disk capability• Manual vs. automated operations (ex. Load balancing)

CLASS is addressing these constraints though several initiatives and standard software releases that will be

discussed as part of the enterprise discussion

System Evolution

Ongoing Initiatives

30

CLASS Projected Archive Volumes

31

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 300.00

50.00

100.00

150.00

200.00

250.00

300.00

TOTAL GOES - R, S, T, U

TOTAL UNDER CONSIDERATION: FUTURE

Model Data (Climate and Weather)

NEXRAD Wx Radar (plus DP & PA)

JPSS Series

NPP

METOP (Current + new launches)

DMSP (Current + new launches)

POES (Current to end of life 18 and 19)

GOES (Current to end of life of 14 and P)

Fiscal Year

Vo

lum

e in

Pet

abyt

es (

PB

)

31

32

NOAA Enterprise Archive System (CLASS)Cumulative Total Volume by Data Type

08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 300.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

NEXRAD

Model

In-Situ

Satellite

FY

Pet

abyt

es

Migration Completed

Projected VolumesActual Volumes

08 09 10 11 12 13 14 15 160.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

NEXRAD

Model

InSitu/Misc.

Satellite

FY

Pet

abyt

es

19.8%

44.7%

27.9%

7.7%

Est. FY 15 Legacy SystemMigration Completed

Projected VolumesActual Volumes

CLASS Projected Ingest and Delivery

33

CLASS Evolution to an Enterprise Archival System

Evolve the existing CLASS hardware and software infrastructure into a distributed, modular, service-oriented architecture • Allow greater flexibility in supporting, to the maximum extent possible, not

only large data arrays from satellite programs but also provide additional archival storage services for all of NOAA’s environmental data that has been approved for archive .

• Working with the NOAA National Data Centers, the new Enterprise Archival Storage architecture will consist of: – Generic ingest services for flexible data acquisition, – Flexible access services using existing community standard, open-source and

emerging technologies such as cloud services,– Standardized metadata repository to support a variety of search and discovery

services and – Long-term, secure archival storage and data management capabilities.

.....continued…

34

CLASS Evolution to an Enterprise Archival System(con’t)

Preservation Planning

Administration

Ingest Access

Data Management

ArchivalStorage

PRODUCER

CONSUMER

requests

results

Present CLASS

Satellite

NOAA Enterprise

ArchivalStorage System

Inge

st

Inte

rfac

e Diss.

Interface

Access Interface

Administration and Preservation

Climate. Gov

Ordering Discovery

Commercial Cloud

Services

Dissemination Services

Access Services

Model

Insitu

Pre-ingest Services

Future CLASS (in green)

Data Gateway

Data Gateway

Data Gateway

Private Cloud

Services

Web Accessible

Folders

Direct Delivery

Admin. Interface

CLASS provides components of the OAIS-RM as services and interfaces. Additional systems and services implemented by the data centers.

35

Satellite

NOAA Enterprise


Inge

st

Inte

rfac

e Diss.

Interface

Access Interface


Climate. Gov

Ordering Discovery

Commercial Cloud

Services


Access Services

Model

Insitu

Pre-ingest Services

Data Gateway

Data Gateway

Data Gateway

Private Cloud

Services

Web Accessible

Folders

Direct Delivery

Admin. Interface

Ongoing InitiativesLeading toward the Enterprise Archival System

Receipt Node/ Gateway• Performs Data Integrity Checks• Provides Temporary Data Storage• Provide translation of data if necessary• Provides a static XML schema for

passing the elements to CLASS• Assigns UUID for preservation in CLASS• Passes metadata information to search

and discovery services

CLASS pivot pointsPre-Ingest Services

(Receipt Node/Gateway)

Common Access Interface(M2M)

Core System Upgrades

Data Managers’ Toolkit

Rules Base Middleware

Common Storage Service(Cloud Pilot Project)

Common Interface Definition

36

Satellite

NOAA Enterprise


Inge

st

Inte

rfac

e Diss.

Interface

Access Interface


Climate. Gov

Ordering Discovery

Commercial Cloud

Services


Access Services

Model

Insitu

Pre-ingest Services

Data Gateway

Data Gateway

Data Gateway

Private Cloud

Services

Web Accessible

Folders

Direct Delivery

Admin. Interface


M2M (Common Access API)• Search, order, and holdings information• RESTful, Asynchronous, interaction with

external systems• Returns all data submitted via Common

Ingest Interface• Metadata publishing to data center

catalogs

CLASS pivot pointsCommon Ingest Interface

(Gateway)







37


Satellite

NOAA Enterprise


Inge

st

Inte

rfac

e Diss.

Interface

Access Interface


Climate. Gov

Ordering Discovery

Commercial Cloud

Services


Access Services

Model

Insitu

Pre-ingest Services

Data Gateway

Data Gateway

Data Gateway

Private Cloud

Services

Web Accessible

Folders

Direct Delivery

Admin. Interface

Common Storage Service• Access area for newly arrived and often

used data sets• Single interface for all access systems• Push once, read many model• Utilizes Cloud Services and Cloud

technologies in hybrid model• Extends CLASS cache out to the

Enterprise


(Gateway)







38


Satellite

NOAA Enterprise


Inge

st

Inte

rfac

e Diss.

Interface

Access Interface


Climate. Gov

Ordering Discovery

Commercial Cloud

Services


Access Services

Model

Insitu

Pre-ingest Services

Data Gateway

Data Gateway

Data Gateway

Private Cloud

Services

Web Accessible

Folders

Direct Delivery

Admin. Interface

Hardware/OS refresh• All CLASS software migrated to Linux

on new hardware• Significant improvements in

performance• HPSS migration will provide more

reliability and flexibility in configuration options


(Gateway)







39

Satellite

NOAA Enterprise

Archival Storage System

Inge

st

Inte

rfac

e Diss.

Interface

Access Interface


Climate. Gov

Ordering Discovery

Commercial Cloud

Services


Access Services

Model

Insitu

Pre-ingest Services

Data Gateway

Data Gateway

Data Gateway

Private Cloud

Services

Web Accessible

Folders

Direct Delivery

Admin. Interface


Common Administration Interface• Single interface data managers

supports:• Stewardship tools• Metadata updating and

versioning• Data holdings monitoring and

statistics


(Gateway)







40

Satellite

NOAA Enterprise Archival

Storage SystemInge

st

Inte

rfac

e Diss.

Interface

Access Interface


Climate. Gov Ordering Discovery

Commercial Cloud Services


Access Services

Model

Insitu

Pre-ingest Services

Data Gateway

Data Gateway

Data Gateway

Private Cloud Services

Web Accessible

Folders

Direct Delivery

Admin. Interface


Rules based middleware• Data Transport

• Moves data objects between systems• Data Information Sharing

• Synchronizes metadata between systems

• Holds data location information • Orchestration

• Implements Enterprise process flow and routes products through Enterprise

• Controlled by data managers and stewards


(Gateway)







41

Satellite

NOAA Enterprise Archival

Storage SystemInge

st

Inte

rfac

e Diss.

Interface

Access Interface


Climate. Gov Ordering Discovery



Access Services

Model

Insitu

Pre-ingest Services

Data Gateway

Data Gateway

Data Gateway

Private Cloud Services

Web Accessible

Folders

Direct Delivery

Admin. Interface


Common Interface Definition• Supports interaction between the

components in the OAIS-RM• Granules• Collections• Browse Images

• Static and well documented• Common Set of Elements• Common Schema • Unique Identifiers


(Gateway)







42

Evolution Roadmap (preliminary) FY2012 FY2013 FY2014 FY2015 FY2016

NODC

NCDC

NGDC

Phase I Phase II Phase III

Cloud

Pilot

Access Path

Metadata

Data NetIRODS

Concurrent CLASS Initiatives

Archive Path


NPP

GCOM-W

Jason On-Hold Programs

NCDCNGDCNODC

AccessDissemination

StewardshipStaging Archive

StorageService

JPSSGOES-R

M2MHPSS

MOB

43

CLASS Evolution to an Enterprise Archival System

CLASS will evolve to:• Implement well-defined interfaces based on industry-standard protocols and best

practices. • Reduce the need for costly custom software development and allow the flexibility to use a multitude of

different COTS discovery tools such as rules-based middleware for data sharing, search and discovery.

• Be scalable to support growth in volume and is extensible • Add functionality and services as new technologies mature and enterprise needs surface.

• Provide a significant and rapid return on investment for NOAA and the Nation• Enable the efficient and inexpensive archive of many NOAA products that are currently awaiting

archival services • Enables the Data Centers to provide new services and products to their customers. .

• Allow the efficient leveraging of other elements of the NOAA Enterprise • Ground system, processing centers, distribution capabilities through generic services and interfaces

44

Other Initiatives that Intersect with CLASS

45

NOAA Enterprise Archival System, Fairmont, WV

NSOF, Suitland MD

NOAA Science Network(N-WAVE)

Users

Development TeamFairmont, WV

Functions

Operations• Ingest• Storage (Disk & Tape)• Public Access

Test and Integration Environment

Satellite Receipt Node/ GatewayDevelopment Environment

Direct Connectivity to:• Environmental Satellite Processing

Center( ESPC) Product Distribution and Access (PDA)

• National Ice Center• NOAA Coastal Watch • JPSS Interface Data Processing Segment (IDPS)• GOES-R Product Distribution (PD)

Notional Fairmont Consolidation Plan System Configuration

Off-site Backup Facility

Direct Connectivity to:• Environmental Satellite Processing

Center( ESPC) Product Distribution and Access (PDA) Backup

• JPSS Interface Data Processing Segment (IDPS) Backup

• GOES-R Product Distribution (PD) Backup

Data Centers, NOAA Gateways


46

Not

iona

lFairmont Consolidation Approach

Phase I – Migrate Development, Test and Integration EnvironmentsPhase II- Establish backup Ingest Capabilities for JPSS, GOES-R, PDAPhase III- Migrate NGDC assets to FairmontPhase IV- Consolidate NCDC assets into Fairmont

Federal Data Center Consolidation

NESDIS Data Center Consolidation

Implementation team established• Begin in 2015 Budget?• Complete by 2023?• Three Data Centers into One?• Final Organizational Structure• Plan not yet approved, in preliminary planning

phases but is a popular idea within NOAA

47

NESDIS Enterprise Ground SystemNOAA/NESDIS assembling EGS to meet its mission• Planning phases

• Technical Reference Model• PDA feasibility Study• Common Storage

• Various approaches under consideration include archive, dissemination

• Driving question- Does CLASS to become the Enterprise Archive part of the EGS?

48

CLASS Programmatic Challenges• Support for the Revision and Rebaseline of Level 1

Requirements– Analysis of Alternatives (Exhibit 300) – Update Full Lifecycle Cost Estimate (GAO Report)

• Clear program governance structure• Adequate Funding for Operations• Rebalance of Requirements/Cost/ Schedule for GOES-R and

JPSS–Budget profile currently does not align with requirements or delivery schedule– Funding Demarcation– Where does the Program’s funding responsibility end?

– Post Launch Support– Operations– Access

49

CLASS Technical ChallengesMetaData- Need to define common elements and format (ISO,

ECHO, Etc.) for ingest and accessInteroperability- Date Information Exchange Language ( can

include meta-data and data formats) for ingest and accessCatalog- Search and discovery services outside of CLASS

domain (Catalog Services for the Web) Data harvesterStorage- External storage for dissemination and access

outside the CLASS domainCon Ops- CLASS has a different role for satellite data and

insitu data

50

Overlap with DAARWG Issues Priority Order from a CLASS Prospective:1. What to archive (Not CLASS but has significant implications)

– What to Archive needs to be coupled with the Budget to Archive

2. Data format (integration and interoperability)- netCDF43. Metadata- Elements and Schema 4. Access

– Storage – Cloud for access (and dissemination)– Access (They do NOT mention Catalogs – maybe that is GEO-IDE????

5. GEO-IDE – Is it still an active viable program

51

comprehensive large array-data stewardship system status presented to daarwg kern witcher class...

Documents

class summary class

evolution of class

class level

class development

class response

sites class

unique data

class definition purpose