1 hycom data service hycom data service an overview ashwanth srinivasan, (fsu) steve hankin...

11

HYCOMHYCOM DataData ServiceService www.hycom.org/dataserverwww.hycom.org/dataserver

An overview An overview

Ashwanth Srinivasan, (FSU)Ashwanth Srinivasan, (FSU)Steve Hankin (NOAA/PMEL)Steve Hankin (NOAA/PMEL)

Contributors:Contributors:Jon Callahan (Mazama Consulting)Jon Callahan (Mazama Consulting)

Roland Schweitzer (Weathertop Consulting), Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL)Ansley Manke (NOAA/PMEL)Jeremy Malczyk (UW/JISAO)Jeremy Malczyk (UW/JISAO)

Peter Cornillon (URI)Peter Cornillon (URI)

22

Data Service was launched in 2002 in Miami and Data Service was launched in 2002 in Miami and served HYCOM Outputs via FTP, LAS and OPeNDAP served HYCOM Outputs via FTP, LAS and OPeNDAP serversservers

Started Serving Near Real Time Atlantic data in 2003 Started Serving Near Real Time Atlantic data in 2003 Hosted about 3 TB of Data until Dec, 2006 in Miami.Hosted about 3 TB of Data until Dec, 2006 in Miami. Jan -2007 – service moved to FSU and hosted on a Jan -2007 – service moved to FSU and hosted on a

100 TB SAN100 TB SAN Currently serving approximately 60 TB of dataCurrently serving approximately 60 TB of data

HYCOM Data Service: History and HYCOM Data Service: History and Current StatusCurrent Status

33

New HardwareNew Hardware

SoftwareSoftware Red Hat LinuxRed Hat Linux Red Hat cluster suite and Red Hat cluster suite and

Global File SystemGlobal File System Apache Web Server Apache Web Server THREDDSTHREDDS LAS ServerLAS Server Vsftpd serverVsftpd server

100 TB Fibre Channel SAN

Three 8 CPU machines

32 GB RAM/machine

44

Operational EnhancementsOperational Enhancements

All components (LAS,OPENDAP,FTP) All components (LAS,OPENDAP,FTP) co-located in the same physical co-located in the same physical machine.machine.

Subset of files are uncompressed for Subset of files are uncompressed for faster accessfaster access

In memory, decompression by using In memory, decompression by using RAM scratch disksRAM scratch disks

High-availability architecture – High-availability architecture – minimal downtime during updatesminimal downtime during updates

55

Currently Available DatasetsCurrently Available Datasets

• Daily outputs from the real time Global Daily outputs from the real time Global HYCOM+NCODA analysis (Apr 2007 –current)HYCOM+NCODA analysis (Apr 2007 –current)

• Global HYCOM+NCODA analysis (2004-2005 Global HYCOM+NCODA analysis (2004-2005 available now; 2005-2007 will be available available now; 2005-2007 will be available shortly)shortly)

• Four years of Global Non-Assimilative Four years of Global Non-Assimilative Simulation (2003-2007) Simulation (2003-2007)

• Near real-time 1/12Near real-time 1/12 Atlantic Ocean prediction Atlantic Ocean prediction system output (June 2003 – Present)system output (June 2003 – Present)

• Monthly mean 1/12Monthly mean 1/12 Pacific Ocean simulation Pacific Ocean simulation output (1978-2003)output (1978-2003)

66

Global HYCOM Data Serving Global HYCOM Data Serving ChallengesChallenges

Very large data/file sizes – each day of output Very large data/file sizes – each day of output is 12-19 G – consists of 26 (15+9) 2d and 3d is 12-19 G – consists of 26 (15+9) 2d and 3d variables (4500 x 3298 x 32 pts)variables (4500 x 3298 x 32 pts)

Curvilinear, staggered and hybrid vertical Curvilinear, staggered and hybrid vertical coordinates makes analysis and visualization coordinates makes analysis and visualization on large files resource intensive and time on large files resource intensive and time consumingconsuming

Need to handle model products in non-Need to handle model products in non-standard native HYCOM format and in standard standard native HYCOM format and in standard format like NetCDF (B.Cs are needed on native format like NetCDF (B.Cs are needed on native grids and in both NetCDF and HYCOM IO grids and in both NetCDF and HYCOM IO formats) formats)

77

HYCOM Ocean Prediction SystemHYCOM Ocean Prediction SystemData FlowData Flow

50 G pulled to FSU~ 2-3 hrs

Conversion to NetCDF andre-gridding to Z levels

THREDDS and

LAS Update

Global run at NAVO daily – 5 day hindcast and 5 day forecast in curvilinear and hybrid coordinates

*Approximately 4-5 hrs of data processing daily

88

Means of Data Access Means of Data Access THREDDSTHREDDS Provides a dataset catalog and OPeNDAP access Provides a dataset catalog and OPeNDAP access

to all datasets.to all datasets.

Live Access ServerLive Access Server VVisualization and downloads in different isualization and downloads in different

formats, batch access etc.formats, batch access etc.

OPeNDAP enabled clientsOPeNDAP enabled clients Choice of common applications or user written Choice of common applications or user written

programs to access dataprograms to access data

FTPFTP NetCDF files for each variable NetCDF files for each variable

99

Provides NetCDF data in Provides NetCDF data in both native HYCOM both native HYCOM (staggered) and regular (staggered) and regular (non-staggered) grids(non-staggered) grids

On-the-fly conversion to On-the-fly conversion to fixed Z levels (but limited fixed Z levels (but limited to pre-configured Z levels)to pre-configured Z levels)

tested HYCOM format IO as tested HYCOM format IO as a product –but needs a product –but needs better integrationbetter integration

Full resolution global Full resolution global outputs not available outputs not available currently via LAS currently via LAS

HYCOM LAS: http://hycom.coaps.fsu.edu/las/

1010

Access to native coordinates and Access to native coordinates and regridded fieldsregridded fields

1111

Hybrid-Z Hybrid-Z regriddingregridding

native-Zfixed-Z

1212

HYDAE model intercomparison

1313

1414

Synthetic Data Used in the Synthetic Data Used in the ExperimentsExperiments

1515

Truth and Forecasts on Day 50 Truth and Forecasts on Day 50

1616

RMS Error in 1-Day ForecastsRMS Error in 1-Day Forecasts

1717

SSH 1-Day Forecast Error SSH 1-Day Forecast Error DistributionDistribution

1818

The Matlab GUI developed for the 1/12o North Atlantic Model Runs has been significantly modified.

Access the GUI from:

http://opendap.org/download/ml-toolbox.html

The OPeNDAP-IDL client is now robust and can be used to access HYCOM data (no GUI yet).

Many modifications to the netCDF-OPeNDAP libraries.

Server4 - supporting GridFTP is now available.

HYCOM specific Matlab tools from the OPeNDAP Developers

1919

The HYCOM Matlab GUIThe HYCOM Matlab GUI

2020

Access the OPeNDAP Matlab GUIs from the Access the OPeNDAP Matlab GUIs from the OPeNDAP download webpage OPeNDAP download webpage

http://opendap.org/downloadhttp://opendap.org/download

Or go directly to the GUI web pageOr go directly to the GUI web page

http://opendap.org/download/ml-http://opendap.org/download/ml-toolbox.htmltoolbox.html

2121

FTP – AccessFTP – AccessFTP://www.hycom.org/datasetsFTP://www.hycom.org/datasets

2323

Access Details for June 2007Access Details for June 2007

2424

Community CollaborationsCommunity Collaborations

Working with John Caron at Unidata Working with John Caron at Unidata on testing the Forecast Aggregation on testing the Forecast Aggregation server with large operational datasetsserver with large operational datasets

Building a HYCOM .[ab] format IO Building a HYCOM .[ab] format IO service provider (ISOP) to be service provider (ISOP) to be incorporated into the JAVA NetCDF 2.2 incorporated into the JAVA NetCDF 2.2 librarylibrary

Partnership with IOOS data Partnership with IOOS data management Caucus management Caucus

2525

Other Addition/ImprovementsOther Addition/Improvements

Forcing dataForcing data Reference observational data sets Reference observational data sets A completely redesigned website is A completely redesigned website is

being planned along the lines of being planned along the lines of GODAE serverGODAE server

Searchable mailing list, FAQ’sSearchable mailing list, FAQ’s Links to other data repositories Links to other data repositories