11
HYCOMHYCOM DataData ServiceService www.hycom.org/dataserverwww.hycom.org/dataserver
An overview An overview
Ashwanth Srinivasan, (FSU)Ashwanth Srinivasan, (FSU)Steve Hankin (NOAA/PMEL)Steve Hankin (NOAA/PMEL)
Contributors:Contributors:Jon Callahan (Mazama Consulting)Jon Callahan (Mazama Consulting)
Roland Schweitzer (Weathertop Consulting), Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL)Ansley Manke (NOAA/PMEL)Jeremy Malczyk (UW/JISAO)Jeremy Malczyk (UW/JISAO)
Peter Cornillon (URI)Peter Cornillon (URI)
22
Data Service was launched in 2002 in Miami and Data Service was launched in 2002 in Miami and served HYCOM Outputs via FTP, LAS and OPeNDAP served HYCOM Outputs via FTP, LAS and OPeNDAP serversservers
Started Serving Near Real Time Atlantic data in 2003 Started Serving Near Real Time Atlantic data in 2003 Hosted about 3 TB of Data until Dec, 2006 in Miami.Hosted about 3 TB of Data until Dec, 2006 in Miami. Jan -2007 – service moved to FSU and hosted on a Jan -2007 – service moved to FSU and hosted on a
100 TB SAN100 TB SAN Currently serving approximately 60 TB of dataCurrently serving approximately 60 TB of data
HYCOM Data Service: History and HYCOM Data Service: History and Current StatusCurrent Status
33
New HardwareNew Hardware
SoftwareSoftware Red Hat LinuxRed Hat Linux Red Hat cluster suite and Red Hat cluster suite and
Global File SystemGlobal File System Apache Web Server Apache Web Server THREDDSTHREDDS LAS ServerLAS Server Vsftpd serverVsftpd server
100 TB Fibre Channel SAN
Three 8 CPU machines
32 GB RAM/machine
44
Operational EnhancementsOperational Enhancements
All components (LAS,OPENDAP,FTP) All components (LAS,OPENDAP,FTP) co-located in the same physical co-located in the same physical machine.machine.
Subset of files are uncompressed for Subset of files are uncompressed for faster accessfaster access
In memory, decompression by using In memory, decompression by using RAM scratch disksRAM scratch disks
High-availability architecture – High-availability architecture – minimal downtime during updatesminimal downtime during updates
55
Currently Available DatasetsCurrently Available Datasets
• Daily outputs from the real time Global Daily outputs from the real time Global HYCOM+NCODA analysis (Apr 2007 –current)HYCOM+NCODA analysis (Apr 2007 –current)
• Global HYCOM+NCODA analysis (2004-2005 Global HYCOM+NCODA analysis (2004-2005 available now; 2005-2007 will be available available now; 2005-2007 will be available shortly)shortly)
• Four years of Global Non-Assimilative Four years of Global Non-Assimilative Simulation (2003-2007) Simulation (2003-2007)
• Near real-time 1/12Near real-time 1/12 Atlantic Ocean prediction Atlantic Ocean prediction system output (June 2003 – Present)system output (June 2003 – Present)
• Monthly mean 1/12Monthly mean 1/12 Pacific Ocean simulation Pacific Ocean simulation output (1978-2003)output (1978-2003)
66
Global HYCOM Data Serving Global HYCOM Data Serving ChallengesChallenges
Very large data/file sizes – each day of output Very large data/file sizes – each day of output is 12-19 G – consists of 26 (15+9) 2d and 3d is 12-19 G – consists of 26 (15+9) 2d and 3d variables (4500 x 3298 x 32 pts)variables (4500 x 3298 x 32 pts)
Curvilinear, staggered and hybrid vertical Curvilinear, staggered and hybrid vertical coordinates makes analysis and visualization coordinates makes analysis and visualization on large files resource intensive and time on large files resource intensive and time consumingconsuming
Need to handle model products in non-Need to handle model products in non-standard native HYCOM format and in standard standard native HYCOM format and in standard format like NetCDF (B.Cs are needed on native format like NetCDF (B.Cs are needed on native grids and in both NetCDF and HYCOM IO grids and in both NetCDF and HYCOM IO formats) formats)
77
HYCOM Ocean Prediction SystemHYCOM Ocean Prediction SystemData FlowData Flow
50 G pulled to FSU~ 2-3 hrs
Conversion to NetCDF andre-gridding to Z levels
THREDDS and
LAS Update
Global run at NAVO daily – 5 day hindcast and 5 day forecast in curvilinear and hybrid coordinates
*Approximately 4-5 hrs of data processing daily
88
Means of Data Access Means of Data Access THREDDSTHREDDS Provides a dataset catalog and OPeNDAP access Provides a dataset catalog and OPeNDAP access
to all datasets.to all datasets.
Live Access ServerLive Access Server VVisualization and downloads in different isualization and downloads in different
formats, batch access etc.formats, batch access etc.
OPeNDAP enabled clientsOPeNDAP enabled clients Choice of common applications or user written Choice of common applications or user written
programs to access dataprograms to access data
FTPFTP NetCDF files for each variable NetCDF files for each variable
99
Provides NetCDF data in Provides NetCDF data in both native HYCOM both native HYCOM (staggered) and regular (staggered) and regular (non-staggered) grids(non-staggered) grids
On-the-fly conversion to On-the-fly conversion to fixed Z levels (but limited fixed Z levels (but limited to pre-configured Z levels)to pre-configured Z levels)
tested HYCOM format IO as tested HYCOM format IO as a product –but needs a product –but needs better integrationbetter integration
Full resolution global Full resolution global outputs not available outputs not available currently via LAS currently via LAS
HYCOM LAS: http://hycom.coaps.fsu.edu/las/
1010
Access to native coordinates and Access to native coordinates and regridded fieldsregridded fields
1111
Hybrid-Z Hybrid-Z regriddingregridding
native-Zfixed-Z
1212
HYDAE model intercomparison
1313
1414
Synthetic Data Used in the Synthetic Data Used in the ExperimentsExperiments
1515
Truth and Forecasts on Day 50 Truth and Forecasts on Day 50
1616
RMS Error in 1-Day ForecastsRMS Error in 1-Day Forecasts
1717
SSH 1-Day Forecast Error SSH 1-Day Forecast Error DistributionDistribution
1818
The Matlab GUI developed for the 1/12o North Atlantic Model Runs has been significantly modified.
Access the GUI from:
http://opendap.org/download/ml-toolbox.html
The OPeNDAP-IDL client is now robust and can be used to access HYCOM data (no GUI yet).
Many modifications to the netCDF-OPeNDAP libraries.
Server4 - supporting GridFTP is now available.
HYCOM specific Matlab tools from the OPeNDAP Developers
1919
The HYCOM Matlab GUIThe HYCOM Matlab GUI
2020
Access the OPeNDAP Matlab GUIs from the Access the OPeNDAP Matlab GUIs from the OPeNDAP download webpage OPeNDAP download webpage
http://opendap.org/downloadhttp://opendap.org/download
Or go directly to the GUI web pageOr go directly to the GUI web page
http://opendap.org/download/ml-http://opendap.org/download/ml-toolbox.htmltoolbox.html
2121
FTP – AccessFTP – AccessFTP://www.hycom.org/datasetsFTP://www.hycom.org/datasets
2323
Access Details for June 2007Access Details for June 2007
2424
Community CollaborationsCommunity Collaborations
Working with John Caron at Unidata Working with John Caron at Unidata on testing the Forecast Aggregation on testing the Forecast Aggregation server with large operational datasetsserver with large operational datasets
Building a HYCOM .[ab] format IO Building a HYCOM .[ab] format IO service provider (ISOP) to be service provider (ISOP) to be incorporated into the JAVA NetCDF 2.2 incorporated into the JAVA NetCDF 2.2 librarylibrary
Partnership with IOOS data Partnership with IOOS data management Caucus management Caucus
2525
Other Addition/ImprovementsOther Addition/Improvements
Forcing dataForcing data Reference observational data sets Reference observational data sets A completely redesigned website is A completely redesigned website is
being planned along the lines of being planned along the lines of GODAE serverGODAE server
Searchable mailing list, FAQ’sSearchable mailing list, FAQ’s Links to other data repositories Links to other data repositories