accessing wave hindcast data - defence research and ... · 1 d:\exe_folder\loadgrib...

$: Accessing wave hindcast data - Defence Research and ... · 1 D:\Exe_Folder\LoadGRIB multi_1.glo_30m.dp.201701.grb2 WWIII 2 D:\Exe_Folder\LoadGRIB multi_1.glo_30m.dp.201702.grb2 WWIII$
Defence Research andDevelopment Canada

Recherche et developpementpour la defense Canada

CAN UNCLASSIFIED

Accessing wave hindcast data

Ian ThompsonJames NickersonDRDC – Atlantic Research Centre

Defence Research and Development CanadaReference DocumentDRDC-RDDC-2019-D105July 2019

CAN UNCLASSIFIED

CAN UNCLASSIFIED

IMPORTANT INFORMATIVE STATEMENTS

This document was reviewed for Controlled Goods by DRDC using the Schedule to the Defence Production Act.

Disclaimer: This publication was prepared by Defence Research and Development Canada an agency of the Department ofNational Defence. The information contained in this publication has been derived and determined through best practice andadherence to the highest standards of responsible conduct of scientific research. This information is intended for the use of theDepartment of National Defence, the Canadian Armed Forces (“Canada") and Public Safety partners and, as permitted, may beshared with academia, industry, Canada’s allies, and the public (“Third Parties"). Any use by, or any reliance on or decisionsmade based on this publication by Third Parties, are done at their own risk and responsibility. Canada does not assume anyliability for any damages or losses which may arise from any use of, or reliance on, the publication.

Endorsement statement: This publication has been published by the Editorial Office of Defence Research and DevelopmentCanada, an agency of the Department of National Defence of Canada. Inquiries can be sent to:[email protected].

© Her Majesty the Queen in Right of Canada, Department of National Defence, 2019

© Sa Majesté la Reine en droit du Canada, Ministère de la Défense nationale, 2019

CAN UNCLASSIFIED

Abstract

A procedure to gain access to wave data from large wave hindcast datasets is described. Thisincludes explaining how to create a database, how to populate the database with wave data intwo binary formats commonly used for these data, and how to extract information from thedatabase. This procedure will make it easier for readers to access freely-available hindcastwave data. Although the procedure has been developed to access NOAA and ECMWFhindcasts, it may be suitable for other hindcasts that use the same data formats.

Résumé

On décrit une procédure d’accès à de l’information sur les vagues provenant de prévisionsa posteriori réalisées à partir de grands ensembles de données. On explique, notamment,comment créer une base de données, comment l’alimenter avec des données sur les vaguesdans deux formats binaires d’usage courant pour ce type de données et comment extraire del’information. Cette procédure aidera les lecteurs qui veulent utiliser les données à accès libresur le sujet. Même si celle-ci a été élaborée spécifiquement pour faciliter l’accès aux prévisionsde la National Oceanic and Atmospheric Administration (NOAA) des États-Unis et à cellesdu Centre européen pour les prévisions météorologiques à moyen terme (CEPMMT), ellepourrait tout de même être utile dans le cas d’autres prévisions comportant des donnéesdans les mêmes formats.

DRDC-RDDC-2019-D105 i

Table of contents

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Résumé . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Table of contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Procedure to access hindcast data . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 Create database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.2 Access hindcast data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.3 Transfer to database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.4 Extract data from database . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

List of acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Annex A: Example procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Annex B: Steps to create a database . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Annex C: Source code for LoadGRIB executable program in C++ . . . . . . . . . . . 12

Annex D: Source code for LoadNetCDF executable program in C++ . . . . . . . . . 20

Annex E: Source code for ExtractWaveData executable program in C++ . . . . . . . 30

ii DRDC-RDDC-2019-D105

1 Introduction

Large sets of accurate wave data covering most of the world’s oceans are currently availablefor analyses related to ship design, operations, or management. Hindcast wave datasets arecalculated based on environmental data and a wind-wave model and are freely availablefrom several organizations. However, these datasets are typically large, which can be achallenge to their effective use. This Reference Document explains how Defence Researchand Development Canada (DRDC) is presently accessing hindcast wave data from reanalysisdatasets. It guides readers to create a database, populate it with wave data in eitherGRIdded Binary (GRIB) [1] or Network Common Data Format (NetCDF) [2] formats, andextract data from the database.

This effort is motivated by interest in improving fatigue damage assessments. Reducing theerror in wave data should reduce the uncertainty in fatigue damage estimates. Also, bycoupling wave data corresponding to the ship’s position at a given time with data from shiptracks (ship speed and heading relative to the wave), the inputs for short-term structuralresponse calculations are available. With this combination, including data on loading condi-tion when it varies, virtual hull monitoring may be possible [3]. The virtual hull monitoringtechnique offers the potential to monitor short-term responses over an extended period with-out additional instrumentation. Initial efforts focus on fatigue damage accumulation, butmonitoring ship motions, and other aspects with transfer functions between the responseof interest and operational and environmental parameters are possible. Although this ap-proach was the motivation behind this effort, large amounts of wave data specific to thetime and location a ship is present can be used in many other types of analyses.

This Reference Document outlines how a database is created and explains how the providedexecutable programs can populate the database and extract relevant data. The approach hasbeen used with the following hindcast datasets: the National Oceanic and Atmospheric Ad-ministration (NOAA) National Weather Service (NWS) National Centers for EnvironmentalPrediction (NCEP) Wavewatch III® 30-year Hindcast Phase 2 [4] (WW3-30 afterwards) andMultigrid Production Hindcast [5] (WW3-Prod hereafter) as well as the European Center forMedium-range Weather Forecasts (ECMWF) ERA5 dataset [6]. The executable programsdiscussed herein will allow readers to access wave data from these hindcasts and the sourcecode may provide a starting point to accessing hindcasts with different output parameters.

The steps required to create and access a wave database are outlined in Section 2; conclusionsand future work are discussed in Section 3. The annexes include an example (Annex A),detailed steps to create a database (Annex B), and the C++ source code for the discussedexecutable programs (Annexes C–E). The executable programs discussed herein will bedistributed to partner organizations, but the provided source code can be re-compiled tocreate them if they are not readily available.

DRDC-RDDC-2019-D105 1

2 Procedure to access hindcast data

This section describes the procedure to create and populate a database and how to extracthindcast wave data from it.

2.1 Create database

The first step in this approach to use wave hindcast data is to create a relational database,which allows values to be stored into one or more tables. Version 11 of the open-sourcePostgreSQL database is used; it may be downloaded at www.postgresql.org/download anddocumentation is available in Reference 7. Since the wave data contains geographic infor-mation, the open-source PostGIS spatial database extender is used. This PostGIS softwareis available at https://postgis.net/install/; Reference 8 contains the documentationand Reference 9 provides a tutorial explaining how to create a new database. The requireddynamic link libraries are provided by installing QGIS, available at download.osgeo.org/qgis/win64/ and selecting the file named QGIS-OSGeo4W-3.0.3-1-Setup-x86_64.exe. Astep by step procedure to create the required database is provided in Annex B. It includesthe required parameter settings to use the executable programs described below.

The executable programs used in this process assume the database is named ‘gisDB’, in-stalled on the user’s local computer, and has ‘drdc’ as a database user and password. Theseparameters are defined in the source code in Annexes C–E. If these parameters are changedin the source code, the programs can be re-compiled with preferred values.

2.2 Access hindcast data

Next, hindcast data files must be downloaded. Data from the WW3-30 hindcast are availableat ftp://polar.ncep.noaa.gov/pub/history/nopp-phase2, WW-Prod data are availableat ftp://polar.ncep.noaa.gov/pub/history/waves/multi_1, and ERA5 data is providedat https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels.The NOAA hindcasts use GRIB files for bulk spectral estimates (significant wave height,peak period, and primary direction) and NetCDF files for partition data (where the wavespectrum is separated into a series of wave systems) and spectral data. The ERA5 hindcastuses GRIB files for all data. The NOAA data come in monthly files with all data for theselected grid whereas the ECMWF site allows users to select which data to download. Thisapproach has been used within DRDC – Atlantic Research Centre to access global modelbulk spectral parameters for all three hindcasts and partition data for the two NOAA hind-casts.

2.3 Transfer to database

An executable program has been created to transfer WW3-30, WW3-Prod, and ERA5GRIB data to the database (LoadGRIB.exe). The program creates data tables, one for

2 DRDC-RDDC-2019-D105

www.postgresql.org/download

https://postgis.net/install/

download.osgeo.org/qgis/win64/


ftp://polar.ncep.noaa.gov/pub/history/nopp-phase2

ftp://polar.ncep.noaa.gov/pub/history/waves/multi_1

https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels

each year, called ‘global_WWIII_data_YEAR’ for the NOAA hindcast GRIB files and‘global_ERA5_data_YEAR’ for the ECMWF hindcast GRIB files. Each table containsthe same eight columns. Their names with explanation and data type in parentheses are:rec_date (record date, timestamp), lat (latitude, real), lon (longitude, real), lon_lat (longi-tude and latitude combination, geometry), wave_height (significant wave height in meters,real), wave_period (peak wave period in seconds, real), wave_dir (predominant wave di-rection in degrees, real), wind_speed (wind speed input to wave model, real). The primarykeys (table columns to uniquely identify records) for the table are rec_date and lon_lat.

GRIB data is inserted into the PostgreSQL database with the command ‘LoadGRIB <GRIBfile name> dataset name’. The <dataset name> refers to the dataset in the database toextract the data from. If the database details are not changed, they are WWIII and ERA5for all of the global_WWIII_data and global_ERA5_data tables, respectively. The C++code used to create the LoadGRIB.exe program is in Annex C. It uses the ecCodes package,available at https://confluence.ecmwf.int/display/ECC/Releases. This program alsouses the Boost C++ library to manipulate date and time; it is available at (https://www.boost.org/users/history/version_1_58_0.html.

The executable program to transfer NetCDF data from the two NOAA hindcasts to thedatabase is called LoadNetCDF.exe. It has been used to import partition data from theNOAA hindcasts into a data table called ‘global_partitions’. At present, only a single tableis used, whereas with the GRIB data, a table is created for each year. This program createssimilar data tables, except they have the following additional columns, with an explanationand data type in parentheses: partition (rank of considered partition, integer), wave_len(wave length in meters, real), spreading_angle (wave spreading angle in degrees, real),and wind_sea_fraction (fraction of total wave energy in considered partition, real). Thepartition column data is added to rec_date and lon_lat as a primary key.

Similar to loading GRIB file data, NetCDF files are imported into the database with thecommand ‘LoadNetCDF <NetCDF file name> table name’. The C++ source code for theLoadNetCDF.exe program is in Annex D. The code uses the NetCDF C++ Interface Guide,available at http://unidata.github.io/netcdf-cxx4/index.html and the Boost C++ li-brary mentioned above.

2.4 Extract data from database

A Structured Query Language (SQL) command is required to extract hindcast data fromthe database. An SQL tutorial is available at Reference 10. The ExtractWaveData exe-cutable program is used to query the database. The command is ‘ExtractWaveData <filename><wave spectra type><round heading value><round speed value><dataset name>’.The <file name> is the ASCII (American Standard Code for Information Interchange) filewith position information. The file contains one line for each record in the following or-der: date, time, latitude, longitude, speed, heading, probability. These entries should betab-delimited. An example is shown in Annex A.


https://confluence.ecmwf.int/display/ECC/Releases

(https://www.boost.org/users/history/version_1_58_0.html

(https://www.boost.org/users/history/version_1_58_0.html

http://unidata.github.io/netcdf-cxx4/index.html

The date should be in the format DD/MM/YYYY, and time defined in UTC (CoordinatedUniversal Time) using a 24-hour clock as HH:MM. Latitude and longitude are defined inthe signed degrees format (DDD.dddd), that is, south latitudes and west longitudes arepreceded with a minus sign and latitudes range from −90° to 90° and longitudes range from−180° to 180°. These programs were created for structural assessments, so values of speed,heading, and probability are required but dummy values may be used.

The <wave spectra type> variable is 1D (one-dimensional); it is included as a variable forongoing work to include partitioned wave data. Rounding of the ship heading is specifiedusing <round heading value>. Headings will be rounded to the nearest multiple of the valueprovided. As an example, if 0.25 was entered, headings would be rounded to the nearestquarter degree. Similarly, ship speed is rounded using <round speed value>. Speed will berounded to the nearest multiple of the value provided. The <dataset name> refers to thedataset in the database to extract the data from. If the database details are not changed,they are WWIII and ERA5 for all of the global_WWIII_data and global_ERA5_datatables, respectively. Partition file data are not exported using the ExtractWaveData exe-cutable program, but the source code in Annex E can be modified to extract these data oradditional parameters.

2.5 Limitations

Although no significant problems with the provided code are known, there are several limi-tations. Because the wave hindcast data files tend to be relatively large, the database sizecan grow quickly. As an example, approximately 20 years of bulk spectral parameter data(about 7 billion records) takes about 3.3 TB (terabyte) of disk space. Partition data requiresseveral times more data because there are commonly more than four partitions in an entry;also for the NOAA hindcasts, it is provided every hour rather than every 3 hours for thebulk spectral estimates. As an indicator of speed, using a database on a 7-year-old workstation can retrieve about 1,000 combinations of bulk spectral parameters per second.


3 Conclusions

The executable programs and procedure outlined in Section 2 provide a method to createa database, populate it, and extract hindcast wave data of interest. This approach shouldallow users to access NOAA and ECMWF hindcasts. Also, if readers would like to accessother hindcast data provided that is formatted differently, the accompanying source codeprovides an example that can be modified. Future work planned within DRDC – AtlanticResearch Centre includes accessing other hindcasts for comparison and extending the use ofpartition data.


References

[1] What is GRIB? (online), https://weather.gc.ca/grib/what_is_GRIB_e.html(Access Date: 17 April 2019).

[2] Network Common Data Form (NetCDF) (online),https://www.unidata.ucar.edu/software/netcdf/ (Access Date: 17 April 2019).

[3] Thompson, I. M. (2018), Virtual Hull Monitoring: continuous fatigue assessmentwithout additional instrumentation, International Journal of Maritime Engineering,160(Part A3), A–293–297.

[4] WAVEWATCH III® 30-year Hindcast Phase 2 (online),http://polar.ncep.noaa.gov/waves/hindcasts/nopp-phase2.php (Access Date: 9October 2017).

[5] WAVEWATCH III® Production Hindcast, Multigrid: Feb 2005 to present (online),http://polar.ncep.noaa.gov/waves/hindcasts/prod-multi_1.php (Access Date:9 October 2017).

[6] ERA5 (online),https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5(Access Date: 17 April 2019).

[7] Documentation (online), https://www.postgresql.org/docs/ (Access Date: 17April 2019).

[8] Documentation (online), https://postgis.net/documentation/ (Access Date: 17April 2019).

[9] Creating a Database (online),https://www.postgresql.org/docs/11/tutorial-createdb.html (Access Date: 17April 2019).

[10] What is SQL? (online), http://www.sqlcourse.com/intro.html (Access Date: 18April 2019).


https://weather.gc.ca/grib/what_is_GRIB_e.html

https://www.unidata.ucar.edu/software/netcdf/

http://polar.ncep.noaa.gov/waves/hindcasts/nopp-phase2.php

http://polar.ncep.noaa.gov/waves/hindcasts/prod-multi_1.php

https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5

https://www.postgresql.org/docs/

https://postgis.net/documentation/

https://www.postgresql.org/docs/11/tutorial-createdb.html

http://www.sqlcourse.com/intro.html

List of acronyms

1D One-dimensionallat Latitudelon Longitudepartition Rank of considered partitionrec_date Record date in hindcastspreading_angle Wave spreading anglewave_dir Predominant wave directionwave_height Significant wave heightwave_len Wave lengthwave_period Peak wave periodwind_sea_fraction fraction of total wave energy in considered partitionwind_speed Input wind speedASCII American Standard Code for Information InterchangeDRDC Defence Research and Development CanadaECMWF European Center for Medium-range Weather ForecastsGRIB GRIdded BinaryNCEP National Centers for Environmental PredictionNOAA National Oceanic and Atmospheric AdministrationNWS National Weather ServiceSQL Structured Query LanguageTB TerabyteUTC Universal Coordinated TimeWW3-30 NOAA NWS NCEP Wavewatch III® 30-year Hindcast Phase 2WW3-Prod NOAA NWS NCEP Wavewatch III® Multigrid Production Hindcast


Annex A Example procedure

Once a database has been created (Section 2.1) and hindcast data has been downloaded(Section 2.2), the wave data can be transferred to the database. This can be done in thecommand prompt for individual files or using a batch file. An example batch file belowloads the global predominant wave direction data for the first three months of 2017 fromthe WW3-Prod hindcast. The ‘glo_30m’ portion refers to the 30-minute global grid.

1 D:\Exe_Folder\LoadGRIB multi_1.glo_30m.dp.201701.grb2 WWIII2 D:\Exe_Folder\LoadGRIB multi_1.glo_30m.dp.201702.grb2 WWIII3 D:\Exe_Folder\LoadGRIB multi_1.glo_30m.dp.201703.grb2 WWIII

Next, data can be extracted provided the position data described in Section 2.4 is availablein a file. An example of the required tab-delimited position data is below.

1 date time lat lon speed heading Probability2 25/02/2003 22:39 46.190 -52.940 10.8 117 0.013 25/02/2003 23:20 46.176 -52.928 10 296 0.014 25/02/2003 23:53 46.220 -52.901 10.3 72 0.015 26/02/2003 00:34 46.225 -52.889 10.7 247 0.016 27/02/2003 01:25 43.551 -55.544 19.9 290 0.01

If the position data shown above is in a file called ‘Position_Data.txt’ in the folder ‘D:\TrialTest’, wave data can be extracted with the batch file below. The reference to 1D indicatesbulk spectral estimates, 1 1 indicates the speed and heading will be rounded to the nearestinteger, and the reference to WWIII indicates the global_WWIII_data will be read. IfERA5 data was loaded, it could be requested by replacing ‘WWIII’ with ‘ERA5’.The resultswill be written to ‘WAVE_DATA_WWIII.dat’.

1 "D:\Exe_Folder\extractWaveData" "D:\Trial Test\Position_Data.txt" 1D 1 1WWIII > WAVE_DATA_WWIII.dat

The output from the batch file listed above is shown below. From left to right the columnsshow: the date, the nearest time entry in the hindcast data, the provided latitude andlongitude, the ship speed and relative heading (rounded to the nearest integer as the batchfile specified), significant wave height, predominant wave direction, and peak period, thesequence of the wave spectra (ws), and the hindcast grid point latitude and longitude wherethe data were selected.

1 Date lat lon speed heading height wave dir peak period ws index hc lathc lon

2 2003-2-25 24:0:0 46.190 -52.940 11 297 3.93 227 12.54 10 46 -533 2003-2-25 24:0:0 46.176 -52.928 10 116 3.93 227 12.54 10 46 -534 2003-2-25 24:0:0 46.220 -52.901 10 252 3.93 227 12.54 10 46 -535 2003-2-26 0:0:0 46.225 -52.889 11 67 3.93 227 12.54 10 46 -536 2003-2-27 0:0:0 43.551 -55.544 20 110 3.95 285 9.77 11 43.5 -55.5


Annex B Steps to create a database

The first step to create a database is to download version 11 of Postgres from www.postgresql.org/download/windows/ and run the installer. Once the software has been installed, it willprompt the user to run Stack Builder to download drivers and extensions, as shown inFigure B.1.

Figure B.1: Installing PostrgreSQL.

The user should select to install Spatial Extension: PostGIS, as shown in Figure B.2.

Figure B.2: Install spatial extension: PostGIS.

The following YouTube video provides a good description on how to install Postgres andthe PostGIS extension: www.youtube.com/watch?v=tTUM9XfDvqk. If Stack Builder fails tolaunch due to firewall restrictions (as occurred within DRDC), users can manually installPostGIS by going to http://download.osgeo.org/postgis/windows/pg11/ and selectingthe postgis-bundle-pg11-2.5.2x64.zip file. This file must be unzipped in the PostgreSQLinstallation directory (e.g., D:/Program Files/PostgreSQL/11).

After installing PostgreSQL and PostGIS, pgAdmin4 should be run. It can be accessedeither through the Windows Start Menu or by running the executable file found in thePostgres bin folder. It will launch a new window in the default browser (Figure B.3) andcreate a task bar icon.


www.postgresql.org/download/windows/

www.postgresql.org/download/windows/

www.youtube.com/watch?v=tTUM9XfDvqk

http://download.osgeo.org/postgis/windows/pg11/

Figure B.3: New pgAdmin4 window.

A new user should be added called ‘drdc’ by right clicking on ‘Login/Group Roles’ andselecting ‘Create Login/Group Role’. On the ‘General’ tab, the name should be set to‘drdc’. On the ‘Definition’ tab, the password should be set to ‘drdc’. On the Privileges tab,‘Can Login?’ should be set to ‘Yes’ and ‘Superuser’ set to ‘Yes’, afterwards, press ‘Save’.

On the hard drive where hindcast data will be stored, users should create a directory called‘gisDBTables’. This hard drive should have ample space for storing large tables. Next,users should create a table space in PostgreSQL by using pgAdmin 4 and right clicking on‘Tablespaces’ in the browser and selecting ‘Create Table Space. . . ’. The name should be setto ‘tabSpace1’ on the ‘General’ tab. On the ‘Definition tab’, ‘Location’ should be set to thelocation of the gisDBTables directory (i.e., D:/gisDBTables). These modifications shouldbe saved.

The next step is to create a new database for the hindcast data by right clicking on‘Databases’ in the pgAdmin4 tool and selecting ‘Create Database. . . ’. On the ‘General’tab, users should set ‘Database’ to ‘gisDB’ and press ‘Save’.

The PostGIS extension must now be added to the database. Users should right click on‘gisDB’ and select ‘Query Tool. . . ’. In the window that opens, users should type ‘createextension postgis’ and press the lightning bolt button to run the SQL command.

Next, users should download and install QGIS by going to download.osgeo.org/qgis/win64/ and selecting the QGIS-OSGeo4W-3.0.3-1-Setup-x86_64.exe file. Once QGIS isinstalled, it must be added to the computer’s environment path variables. Edit the ‘Path’system variable and add ‘D:\QGIS 2.18\bin;D:\QGIS’ to the start of the string. Of course,this should be modified if QGIS is installed in a different location.




The eccodes package (version eccodes-2.8.0-Source.tar.gz) should be downloaded from confluence.ecmwf.int/display/ECC/Releases/ and decompressed. Afterwards, a new environmentvariable called ‘ECCODES_DEFINITION_PATH’ should be created with a value of ‘D:\eccodes-2.8.0-Source\definitions’ where ‘D:’ is the drive where the eccodes package was extracted.The executable programs to load and extract hindcast data to and from the database (i.e.,LoadGrib.exe, LoadNetCDF.exe, and extractWaveData.exe) can now be run.


confluence.ecmwf.int/display/ECC/Releases/

confluence.ecmwf.int/display/ECC/Releases/

Annex C Source code for LoadGRIB executableprogram in C++

1 /*2 * C Implementation: grib_get_keys3 *4 * Description: how to get values using keys from GRIB messages5 *6 */7 #include <stdio.h>8 #include <sstream>9 #include <boost/date_time/local_time/local_date_time.hpp>

10 #include <boost/date_time/posix_time/posix_time_io.hpp>11 #include "eccodes.h"12 #include <libpq-fe.h>13

14 #define MAX_KEY_LEN 25515 #define MAX_VAL_LEN 102416 #define PG_HOST "127.0.0.1"17 #define PG_USER "drdc"18 #define PG_DB "gisDB"19 #define PG_PASS "drdc"20 #define PG_PORT 543221

22 using namespace boost::posix_time;23

24 void usage(char* prog)25 {26 printf("Usage: %s grib_file table_type<WWIII ERA5>\n",prog);27 exit(1);28 }29

30 static void exit_nicely(PGconn *conn, PGresult *res)31 {32 PQclear(res);33 PQfinish(conn);34 exit(1);35 }36

37 int main(int argc, char** argv)38 {39 clock_t end_time, first_time;40 int err = 0;


41 char conninfo[250];42 PGconn *conn = NULL;43 PGresult *pgres = NULL;44 bool mediterraneanOnly = false;45 const double maxLat = 67.0;46 const double minLat = 10.0;47 const double maxLon = 60.0;48 const double minLon = -7.0;49

50 std::stringstream dbTableName;51

52 if ( argc == 4 && strcmp(argv[3], "med") == 0 )53 mediterraneanOnly = true;54 else if (argc != 3)55 usage(argv[0]);56

57 if ( strcmp(argv[2], "WWIII") == 0 )58 dbTableName << "global_WWIII_data_";59 else if ( strcmp(argv[2], "ERA5") == 0 )60 dbTableName << "global_ERA5_data_";61 else62 {63 usage(argv[0]);64 exit_nicely(conn,pgres);65 }66

67 FILE* in = NULL;68 const char* filename=_strdup(argv[1]);69

70 first_time = clock(); // Start time for reading the data from the cdf file71

72 // Make a connection to the database73 sprintf_s(conninfo, "user=%s password=%s dbname=%s hostaddr=%s port=%d",

PG_USER, PG_PASS, PG_DB, PG_HOST, PG_PORT);74 conn = PQconnectdb(conninfo);75

76 // Check to see that the back end connection was successfully made77 if (PQstatus(conn) != CONNECTION_OK)78 {79 fprintf(stderr, "Connection to database failed: %s", PQerrorMessage(conn));80 exit_nicely(conn,pgres);81 }82

83 fopen_s(&in, filename,"rb");


84 if(!in) {85 printf("ERROR: unable to open file %s\n",filename);86 return 1;87 }88

89 /* Choose a namespace. E.g. "ls", "time", "parameter", "geography","statistics" */

90 char* name_space=0;91

92 /* Iterator on lat/lon/values.*/93 codes_iterator* iter=NULL;94

95 codes_handle *h = NULL;96 double lat,lon,value,missingValue=0;97 unsigned long key_iterator_filter_flags=GRIB_KEYS_ITERATOR_ALL_KEYS;98

99 // Get the year from the data file100 int year;101

102 /* Loop on all the messages in a file.*/103 while ( (h=codes_handle_new_from_file(0, in, PRODUCT_GRIB, &err)) != NULL )104 {105 grib_keys_iterator* kiter=NULL;106 kiter=grib_keys_iterator_new(h,key_iterator_filter_flags,name_space);107

108 char date[15];109 while(grib_keys_iterator_next(kiter))110 {111 const char* name = grib_keys_iterator_get_name(kiter);112 size_t vlen=MAX_VAL_LEN;113 char value[MAX_VAL_LEN];114

115 if( strcmp(name, "dataDate") == 0 )116 {117 CODES_CHECK(grib_get_string(h,name,value,&vlen),name);118 strcpy_s( date, value);119 }120 }121

122 // Extract the grib date and time123 int month, day;124 sscanf_s(date,"%4d%2d%2d",&year,&month,&day);125 break;126 }


127 fclose(in);128

129 dbTableName << year;130

131 // Create a table to store the data132 std::stringstream createTable;133 createTable << "CREATE TABLE IF NOT EXISTS " << dbTableName.str() << "

(rec_date TIMESTAMP, lat REAL, lon REAL, lon_lat GEOMETRY(POINT, 4326),wave_height REAL, wave_period REAL, wave_dir REAL, wind_speed REAL,PRIMARY KEY(rec_date, lon_lat)) TABLESPACE \"tabSpace1\"" << std::endl;

134

135 pgres = PQexec( conn, createTable.str().c_str() );136 if(PQresultStatus(pgres) != PGRES_COMMAND_OK)137 {138 fprintf(stderr, "CREATE TABLE failed: %s", PQerrorMessage(conn));139 exit_nicely(conn,pgres);140 }141 PQclear(pgres);142

143

144 fopen_s(&in, filename,"rb");145 if(!in) {146 printf("ERROR: unable to open file %s\n",filename);147 return 1;148 }149

150 /* Loop on all the messages in a file.*/151 while ( (h=codes_handle_new_from_file(0, in, PRODUCT_GRIB, &err)) != NULL )152 {153 std::stringstream columnName;154 grib_keys_iterator* kiter=NULL;155 kiter=grib_keys_iterator_new(h,key_iterator_filter_flags,name_space);156

157 int hours;158 char date[15];159 while(grib_keys_iterator_next(kiter))160 {161 const char* name = grib_keys_iterator_get_name(kiter);162 size_t vlen=MAX_VAL_LEN;163 char value[MAX_VAL_LEN];164

165 if( strcmp(name, "dataDate") == 0 )166 {167 CODES_CHECK(grib_get_string(h,name,value,&vlen),name);


168 strcpy_s( date, value);169 }170 else if( strcmp(name, "startStep") == 0 )171 {172 CODES_CHECK(grib_get_string(h,name,value,&vlen),name);173 hours = atoi(value);174 }175 else if( strcmp(name, "name") == 0 )176 {177 CODES_CHECK(grib_get_string(h,name,value,&vlen),name);178 if( strcmp(value, "Mean wave direction") == 0 )179 columnName << "wave_dir";180 else if( strcmp(value, "Peak wave period") == 0 )181 columnName << "wave_period";182 else if( strcmp(value, "Significant height of combined wind waves and

swell") == 0 )183 columnName << "wave_height";184 else if( strcmp(value, "Primary wave direction") == 0 )185 columnName << "wave_dir";186 else if( strcmp(value, "Peak wave mean period") == 0 )187 columnName << "wave_period";188 else if( strcmp(value, "Primary wave mean period") == 0 )189 columnName << "wave_period";190 else if( strcmp(value, "U component of wind") == 0 )191 columnName << "wind_speed";192 }193 }194

195 // Extract the grib date and time196 int year, month, day;197 sscanf_s(date,"%4d%2d%2d",&year,&month,&day);198 ptime grib_date( boost::gregorian::date(year, month, day),

boost::posix_time::hours(0)+minutes(0)+seconds(0) );199 grib_date += boost::posix_time::hours( hours );200 std::cout << grib_date << " " << columnName.str() << std::endl;201

202 /* Check of errors after reading a message. */203 if (err != GRIB_SUCCESS) GRIB_CHECK(err,0);204

205 /* Get the double representing the missing value in the field. */206 CODES_CHECK(grib_get_double(h,"missingValue",&missingValue),0);207

208 /* A new iterator on lat/lon/values is created from the message handle h. */209 iter=grib_iterator_new(h,0,&err);


210 if (err != GRIB_SUCCESS) GRIB_CHECK(err,0);211

212 /* start a transaction block */213 pgres = PQexec(conn, "BEGIN");214 if (!pgres || PQresultStatus(pgres) != PGRES_COMMAND_OK)215 {216 fprintf(stderr, "BEGIN command failed\n");217 exit_nicely(conn,pgres);218 }219 PQclear(pgres);220

221 std::stringstream querryStr;222 querryStr << "INSERT INTO " << dbTableName.str() << " (rec_date, lat, lon,

lon_lat, " << columnName.str() << ") VALUES ";223 bool firstRecord = true;224

225 /* Loop on all the lat/lon/values. */226 while(grib_iterator_next(iter,&lat,&lon,&value))227 {228 if(value == missingValue)229 continue;230

231 std::stringstream latStr, lonStr, lon_latStr, dirStr;232 if( lon > 180.0 )233 lon = lon - 360.0;234

235 latStr << lat;236 lonStr << lon;237 lon_latStr << "ST_SetSRID(ST_MakePoint(" << lon << ", " << lat << "),

4326)";238 dirStr << value;239

240 char* params[5];241 params[0] = new char[to_simple_string(grib_date).length() + 1];242 params[1] = new char[latStr.str().length() + 1];243 params[2] = new char[lonStr.str().length() + 1];244 params[3] = new char[lon_latStr.str().length() + 1];245 params[4] = new char[dirStr.str().length() + 1];246

247 strcpy_s(params[0], to_simple_string(grib_date).length() + 1,to_simple_string(grib_date).c_str());

248 strcpy_s(params[1], latStr.str().length() + 1, latStr.str().c_str());249 strcpy_s(params[2], lonStr.str().length() + 1, lonStr.str().c_str());


250 strcpy_s(params[3], lon_latStr.str().length() + 1,lon_latStr.str().c_str());

251 strcpy_s(params[4], dirStr.str().length() + 1, dirStr.str().c_str());252

253 if( grib_iterator_has_next(iter) && !firstRecord )254 querryStr << ", ";255 else256 firstRecord = false;257

258 querryStr << "('" << params[0] << "', " << params[1] << ", " << params[2]<< ", " << params[3] << ", " << params[4] << ")";

259

260 delete params[0];261 delete params[1];262 delete params[2];263 delete params[3];264 delete params[4];265 }266

267 if( strcmp(columnName.str().c_str(), "wave_height") == 0 )268 querryStr << " ON CONFLICT (rec_date, lon_lat) DO UPDATE SET (wave_height,

wave_period, wave_dir, wind_speed) = (EXCLUDED.wave_height, " <<dbTableName.str() << ".wave_period, " << dbTableName.str() <<".wave_dir, " << dbTableName.str() << ".wind_speed)";

269 else if( strcmp(columnName.str().c_str(), "wave_period") == 0 )270 querryStr << " ON CONFLICT (rec_date, lon_lat) DO UPDATE SET (wave_height,

wave_period, wave_dir, wind_speed) = (" << dbTableName.str() <<".wave_height, EXCLUDED.wave_period, " << dbTableName.str() <<".wave_dir, " << dbTableName.str() << ".wind_speed)";

271 else if( strcmp(columnName.str().c_str(), "wave_dir") == 0 )272 querryStr << " ON CONFLICT (rec_date, lon_lat) DO UPDATE SET (wave_height,

wave_period, wave_dir, wind_speed) = (" << dbTableName.str() <<".wave_height, " << dbTableName.str() << ".wave_period,EXCLUDED.wave_dir, " << dbTableName.str() << ".wind_speed)";

273 else if( strcmp(columnName.str().c_str(), "wind_speed") == 0 )274 querryStr << " ON CONFLICT (rec_date, lon_lat) DO UPDATE SET (wave_height,

wave_period, wave_dir, wind_speed) = (" << dbTableName.str() <<".wave_height, " << dbTableName.str() << ".wave_period, " <<dbTableName.str() << ".wave_dir, EXCLUDED.wind_speed)";

275

276 /* At the end the iterator is deleted to free memory. */277 grib_iterator_delete(iter);278

279 /* At the end the grib_handle is deleted to free memory. */


280 grib_handle_delete(h);281

282 pgres = PQexec(conn, querryStr.str().c_str());283 if(PQresultStatus(pgres) != PGRES_COMMAND_OK)284 {285 fprintf(stderr, "Insert failed: %s", PQerrorMessage(conn));286 exit_nicely(conn,pgres);287 }288 PQclear(pgres);289

290 /* commit the transaction */291 pgres = PQexec(conn, "COMMIT");292 PQclear(pgres);293 }294

295 /* close the connection to the database and cleanup */296 PQfinish(conn);297 fclose(in);298

299 end_time = clock() - first_time;300 printf ("Total time = %f

hours\n",((float)end_time)/CLOCKS_PER_SEC/60.0/60.0);301 printf("*** SUCCESS reading file: %s\n", filename);302

303 return 0;304 }


Annex D Source code for LoadNetCDF executableprogram in C++

1 #include <stdio.h>2 #include <sstream>3 #include <iostream>4 #include <string.h>5 #include <time.h>6 #include <netcdf.h>7 #include <libpq-fe.h>8 #include <boost/date_time/local_time/local_date_time.hpp>9 #include <boost/date_time/posix_time/posix_time_io.hpp>

10

11 #define DAT_NAME "date"12 #define PAR_NAME "partition"13 #define LAT_NAME "latitude"14 #define LON_NAME "longitude"15 #define DEPTH_NAME "depth"16 #define WIND_SPEED_NAME "wind_speed"17 #define SIG_WAVE_HEIGHT_NAME "significant_wave_height"18 #define PEAK_PERIOD_NAME "peak_period"19 #define WAVE_LENGTH_NAME "wavelength"20 #define WAVE_DIRECTION_NAME "wave_direction"21 #define SPREADING_ANGLE_NAME "direction_spreading"22 #define WIND_SEA_FRACTION_NAME "wind_sea_fraction"23

24 // defines for POSTGRES25 #define MAX_KEY_LEN 25526 #define MAX_VAL_LEN 102427 #define PG_HOST "127.0.0.1"28 #define PG_USER "drdc"29 #define PG_DB "gisDB"30 #define PG_PASS "drdc"31 #define PG_PORT 543232

33 /* Handle errors by printing an error message and exiting with a non-zerostatus. */

34 #define ERR(e) {printf("Error: %s\n", nc_strerror(e)); return 2;}35

36 using namespace std;37 using namespace boost::posix_time;38

39 void usage(char* prog)


40 {41 printf("Usage: %s <NetCDF Filename>\n",prog);42 exit(1);43 }44


52 int main(int argc, char* argv[])53 {54 clock_t start_time, end_time, first_time;55 int ncid;56 int lat_varid, lon_varid, date_varid, part_varid, depth_varid, ws_varid,

swh_varid, pp_varid, wl_vardid, wd_varid, sa_varid, wsf_varid;57 char var_name[NC_MAX_NAME];58

59 char conninfo[250];60 PGconn *conn = NULL;61 PGresult *pgres = NULL;62 size_t num_lats=0, num_lons=0, num_dates=0, num_parts=0;63 int *partitions;64

65 /* These program variables hold the latitudes, longitudes, and dates. */66 float *lats, *lons, *dates;67

68 if (argc != 2) usage(argv[0]);69

70 const char* filename=_strdup(argv[1]);71

72 /* Error handling. */73 int retval;74

75 /* Open the file. */76 if ( retval = nc_open(filename, NC_NOWRITE, &ncid) )77 ERR(retval);78

79 /* Get the varids of the variables. */80 if( retval = nc_inq_varid(ncid, DAT_NAME, &date_varid) )81 ERR(retval);82 if( retval = nc_inq_varid(ncid, PAR_NAME, &part_varid) )


83 ERR(retval);84 if( retval = nc_inq_varid(ncid, LAT_NAME, &lat_varid) )85 ERR(retval);86 if( retval = nc_inq_varid(ncid, LON_NAME, &lon_varid) )87 ERR(retval);88 if (retval = nc_inq_varid(ncid, DEPTH_NAME, &depth_varid) )89 ERR(retval);90 if (retval = nc_inq_varid(ncid, WIND_SPEED_NAME, &ws_varid) )91 ERR(retval);92 if (retval = nc_inq_varid(ncid, SIG_WAVE_HEIGHT_NAME, &swh_varid) )93 ERR(retval);94 if (retval = nc_inq_varid(ncid, PEAK_PERIOD_NAME, &pp_varid) )95 ERR(retval);96 if (retval = nc_inq_varid(ncid, WAVE_LENGTH_NAME, &wl_vardid) )97 ERR(retval);98 if (retval = nc_inq_varid(ncid, WAVE_DIRECTION_NAME, &wd_varid) )99 ERR(retval);

100 if (retval = nc_inq_varid(ncid, SPREADING_ANGLE_NAME, &sa_varid) )101 ERR(retval);102 if (retval = nc_inq_varid(ncid, WIND_SEA_FRACTION_NAME, &wsf_varid) )103 ERR(retval);104

105 // Get the number of each variable106 if( retval = nc_inq_dim(ncid, date_varid, var_name, &num_dates) )107 ERR(retval);108 if( retval = nc_inq_dim(ncid, part_varid, var_name, &num_parts) )109 ERR(retval);110 if( retval = nc_inq_dim(ncid, lat_varid, var_name, &num_lats) )111 ERR(retval);112 if( retval = nc_inq_dim(ncid, lon_varid, var_name, &num_lons) )113 ERR(retval);114

115 // Allocate memory for the lats and lons116 dates = new float[num_dates];117 lats = new float[num_lats];118 lons = new float[num_lons];119 partitions = new int [num_parts];120

121 /* Read the dates and coordinate variable data. */122 if ( retval = nc_get_var_float(ncid, date_varid, dates) )123 ERR(retval);124 if ( retval = nc_get_var_int(ncid, part_varid, partitions) )125 ERR(retval);126 if ( retval = nc_get_var_float(ncid, lat_varid, lats) )


127 ERR(retval);128 if ( retval = nc_get_var_float(ncid, lon_varid, lons) )129 ERR(retval);130

131 /* The start and count arrays will tell the netCDF library where to readour data. */

132 size_t start2D[2], count2D[2];133

134 /* Read the data. Since we know the contents of the file we know135 * that the data arrays in this program are the correct size to136 * hold one time step. */137 count2D[0] = num_lats;138 count2D[1] = num_lons;139 start2D[0] = 0;140 start2D[1] = 0;141

142 float *depths = new float[num_lats*num_lons];143 if ( retval = nc_get_vara_float(ncid, depth_varid, start2D, count2D,

depths) )144 ERR(retval);145 delete depths;146

147 // convert number of days since January 1, 1970 to number of seconds148 ptime *converted_dates = new ptime[num_dates];149 string *dateStr = new string[num_dates];150 for( int i=0; i<num_dates; i++ )151 {152 float days_since_1970 = dates[i];153 time_t seconds_since_1970 = time_t(days_since_1970*24.0*60.0*60.0);154 ptime date = from_time_t(seconds_since_1970);155 converted_dates[i] = date;156

157 int day = converted_dates[i].date().day();158 int month = converted_dates[i].date().month();159 int year = converted_dates[i].date().year();160 int hours = converted_dates[i].time_of_day().hours();161 int minutes = converted_dates[i].time_of_day().minutes();162 if(minutes > 30 )163 hours++;164

165 if( hours == 24 )166 hours = 0;167


168 dateStr[i] = to_string(year) + '-' + to_string(month) + '-' +to_string(day) + ' ' + to_string(hours) + ":0:0";

169 }170




182 // Create a table to store the data183 std::stringstream createTable;184 createTable << "CREATE TABLE IF NOT EXISTS global_partitions" <<185 " (rec_date TIMESTAMP, lat REAL, lon REAL, lon_lat GEOMETRY(POINT, 4326),

partition INTEGER, wave_height REAL, peak_period REAL, wave_lengthREAL, wave_dir REAL, spreading_angle REAL, wind_sea_fraction REAL,PRIMARY KEY(rec_date, lon_lat, partition))" << std::endl;

186

187 pgres = PQexec( conn, createTable.str().c_str() );188 if(PQresultStatus(pgres) != PGRES_COMMAND_OK)189 {190 fprintf(stderr, "CREATE TABLE failed: %s", PQerrorMessage(conn));191 exit_nicely(conn,pgres);192 }193 PQclear(pgres);194

195 /* The start and count arrays will tell the netCDF library where to readour data. */

196 size_t start4D[4], count4D[4];197

198 printf("Begin Inserting Data\n");199 float *sig_wave_height = new float[num_lons*num_lats];200 float *peak_period = new float[num_lons*num_lats];201 float *wave_length = new float[num_lons*num_lats];202 float *wave_dir = new float[num_lons*num_lats];203 float *spreading_angle = new float[num_lons*num_lats];204 float *wind_sea_fraction = new float[num_lons*num_lats];205


206 start_time = clock(); // Start time for reading the data from the cdf file207 first_time = start_time;208 std::stringstream querrySTR;209 querrySTR << "INSERT INTO global_partitions (rec_date, lat, lon, lon_lat,

partition, wave_height, peak_period, wave_length, wave_dir,spreading_angle, wind_sea_fraction) VALUES ";

210 for( int day=0; day<num_dates; day++ )211 {212 cout << "Day: " << day << endl;213 for(int par=0; par<num_parts; par++)214 {215 int num_records = 0;216 count4D[0] = 1;217 count4D[1] = 1;218 count4D[2] = num_lats;219 count4D[3] = num_lons;220 start4D[0] = day;221 start4D[1] = par;222 start4D[2] = 0/*lat*/;223 start4D[3] = 0/*lon*/;224

225 if ( retval = nc_get_vara_float(ncid, swh_varid, start4D, count4D,sig_wave_height ))

226 ERR(retval);227 if ( retval = nc_get_vara_float(ncid, pp_varid, start4D, count4D,

peak_period ))228 ERR(retval);229 if ( retval = nc_get_vara_float(ncid, wl_vardid, start4D, count4D,

wave_length ))230 ERR(retval);231 if ( retval = nc_get_vara_float(ncid, wd_varid, start4D, count4D, wave_dir

))232 ERR(retval);233 if ( retval = nc_get_vara_float(ncid, sa_varid, start4D, count4D,

spreading_angle ))234 ERR(retval);235 if ( retval = nc_get_vara_float(ncid, wsf_varid, start4D, count4D,

wind_sea_fraction ))236 ERR(retval);237

238 for (size_t lat = 0; lat < num_lats; lat++)239 {240 for (size_t lon = 0; lon < num_lons; lon++)241 {


242 if(sig_wave_height[lon + lat * num_lons] != -999.0 && peak_period[lon + lat* num_lons] != -999.0 && wave_length[lon + lat * num_lons] != -999.0 &&wave_dir[lon + lat * num_lons] != -999.0 && spreading_angle[lon + lat *num_lons] != -999.0 && wind_sea_fraction[lon + lat * num_lons] !=-999.0)

243 {244 std::stringstream latStr, lonStr, lon_latStr, parStr, sig_wave_heightStr,

peak_periodStr, wave_lengthStr, wave_dirStr, spreading_angleStr,wind_sea_fractionStr;

245 float latitude = lats[lat];246 float longitude = lons[lon];247 if( longitude > 180.0 )248 longitude = longitude - (float)360.0;249

250 latStr << latitude;251 lonStr << longitude;252 lon_latStr << "ST_SetSRID(ST_MakePoint(" << longitude << ", " << latitude

<< "), 4326)";253 parStr << partitions[par];254 sig_wave_heightStr << sig_wave_height[lon + lat * num_lons];255 peak_periodStr << peak_period[lon + lat * num_lons];256 wave_lengthStr << wave_length[lon + lat * num_lons];257 wave_dirStr << wave_dir[lon + lat * num_lons];258 spreading_angleStr << spreading_angle[lon + lat * num_lons];259 wind_sea_fractionStr << wind_sea_fraction[lon + lat * num_lons];260

261 char* params[11];262 params[0] = new char[to_simple_string(converted_dates[day]).length() + 1];263 params[1] = new char[latStr.str().length() + 1];264 params[2] = new char[lonStr.str().length() + 1];265 params[3] = new char[lon_latStr.str().length() + 1];266 params[4] = new char[parStr.str().length() + 1];267 params[5] = new char[sig_wave_heightStr.str().length() + 1];268 params[6] = new char[peak_periodStr.str().length() + 1];269 params[7] = new char[wave_lengthStr.str().length() + 1];270 params[8] = new char[wave_dirStr.str().length() + 1];271 params[9] = new char[spreading_angleStr.str().length() + 1];272 params[10] = new char[wind_sea_fractionStr.str().length() + 1];273

274 strcpy_s(params[1] , latStr.str().length() + 1, latStr.str().c_str());275 strcpy_s(params[2] , lonStr.str().length() + 1, lonStr.str().c_str());276 strcpy_s(params[3] , lon_latStr.str().length() + 1,

lon_latStr.str().c_str());277 strcpy_s(params[4] , parStr.str().length() + 1, parStr.str().c_str());


278 strcpy_s(params[5] , sig_wave_heightStr.str().length() + 1,sig_wave_heightStr.str().c_str());

279 strcpy_s(params[6] , peak_periodStr.str().length() + 1,peak_periodStr.str().c_str());

280 strcpy_s(params[7] , wave_lengthStr.str().length() + 1,wave_lengthStr.str().c_str());

281 strcpy_s(params[8] , wave_dirStr.str().length() + 1,wave_dirStr.str().c_str());

282 strcpy_s(params[9] , spreading_angleStr.str().length() + 1,spreading_angleStr.str().c_str());

283 strcpy_s(params[10], wind_sea_fractionStr.str().length() + 1,wind_sea_fractionStr.str().c_str());

284

285 querrySTR << "('" << dateStr[day] << "', " << params[1] << ", " <<params[2] << ", " << params[3] << ", " << params[4] << ", " <<params[5] << ", " << params[6] << ", " << params[7] << ", " <<params[8] << ", " << params[9] << ", " << params[10] << "),";

286 num_records++;287

288 delete params[0];289 delete params[1];290 delete params[2];291 delete params[3];292 delete params[4];293 delete params[5];294 delete params[6];295 delete params[7];296 delete params[8];297 delete params[9];298 delete params[10];299 }300 }301 }302

303 if( !num_records )304 continue;305

306 cout << "Number of records: " << num_records << endl;307

308 end_time = clock() - start_time; // End time for reading the data from thecdf file

309 printf ("It took %f minutes to read thedata.\n",((float)end_time)/CLOCKS_PER_SEC/60.0);

310


311 querrySTR.seekp(-1, std::ios_base::end);312

313 querrySTR << " ON CONFLICT (rec_date, lon_lat, partition) DO UPDATE SET(wave_height, peak_period, wave_length, wave_dir, spreading_angle,wind_sea_fraction) = (EXCLUDED.wave_height, EXCLUDED.peak_period,EXCLUDED.wave_length, EXCLUDED.wave_dir, EXCLUDED.spreading_angle,EXCLUDED.wind_sea_fraction)";

314

315 start_time = clock();316 pgres = PQexec(conn, querrySTR.str().c_str());317 if(PQresultStatus(pgres) != PGRES_COMMAND_OK)318 {319 fprintf(stderr, "Insert failed: %s", PQerrorMessage(conn));320 cout << querrySTR.str() << endl;321 exit_nicely(conn,pgres);322 }323 PQclear(pgres);324

325 end_time = clock() - start_time;326 printf ("It took %f minutes to insert into the

database.\n",((float)end_time)/CLOCKS_PER_SEC/60.0);327

328 start_time = clock();329 querrySTR.str("");330 querrySTR.clear();331 querrySTR << "INSERT INTO global_partitions (rec_date, lat, lon, lon_lat,

partition, wave_height, peak_period, wave_length, wave_dir,spreading_angle, wind_sea_fraction) VALUES ";

332 }333 }334

335 /* close the connection to the database and cleanup */336 PQfinish(conn);337

338 /* Close the file. */339 if ((retval = nc_close(ncid)))340 ERR(retval);341

342 delete sig_wave_height;343 delete peak_period;344 delete wave_length;345 delete wave_dir;346 delete spreading_angle;347 delete wind_sea_fraction;


348

349 delete dates;350 delete converted_dates;351 delete partitions;352 delete lats;353 delete lons;354

355 end_time = clock() - first_time;356 printf ("Total time = %f

hours\n",((float)end_time)/CLOCKS_PER_SEC/60.0/60.0);357 printf("*** SUCCESS reading file: %s\n", filename);358 return 0;359 }


Annex E Source code for ExtractWaveData executableprogram in C++

1 // extractWaveData.cpp : Defines the entry point for the consoleapplication.

2 //3

4 #include <stdio.h>5 #include <sstream>6 #include <iostream>7 #include <fstream>8 #include <string>9 #include <boost/math/constants/constants.hpp>

10 #include <boost/date_time/local_time/local_date_time.hpp>11 #include <boost/date_time/posix_time/posix_time_io.hpp>12 #include <libpq-fe.h>13

14 #define PG_HOST "127.0.0.1"15 #define PG_USER "drdc"16 #define PG_DB "gisDB"17 #define PG_PASS "drdc"18 #define PG_PORT 543219 #define M_PI boost::math::double_constants::pi20

21 using namespace std;22 using namespace boost::posix_time;23 using namespace boost::gregorian;24

25 struct position26 {27 double lat;28 double lon;29 double distance;30 };31

32 struct wave_spectra_Bretschneider33 {34 double hSig;35 double wavePeakPeriod;36 int wsIndex;37 };38

39 struct wave_spectra_Kumar


40 {41 int wsIndex;42 int numPartitions;43 bool useCompassHeadings;44 double shipDirectionNorthDeg;45 vector<double> sigWaveHeight;46 vector<double> wavePeriod;47 vector<double> spreadAngleDeg;48 vector<double> spreadExp;49 vector<double> waveDir;50 vector<double> waveHeadingsDegs;51 };52

53 void usage(char* prog)54 {55 printf("Usage: %s GPS data file>\n",prog);56 exit(1);57 }58


66 std::vector<std::string> split(const std::string& s, char delimiter)67 {68 std::vector<std::string> tokens;69 std::string token;70 std::istringstream tokenStream(s);71 while (std::getline(tokenStream, token, delimiter))72 {73 tokens.push_back(token);74 }75 return tokens;76 }77

78 int insertUniqueWaveSpectra( wave_spectra_Bretschneider &ws,vector<wave_spectra_Bretschneider> &wsList )

79 {80 int i=0;81 bool found = false;82 for( i=0; i<wsList.size(); i++ )


83 {84 wave_spectra_Bretschneider wsTmp = wsList.at(i);85 if( ws.hSig == wsTmp.hSig && ws.wavePeakPeriod == wsTmp.wavePeakPeriod )86 {87 found = true;88 break;89 }90 }91

92 if( !found )93 {94 ws.wsIndex = (int)wsList.size();95 wsList.push_back( ws );96 return ws.wsIndex;97 }98 else99 {

100 ws.wsIndex = i;101 return ws.wsIndex;102 }103 }104

105 int insertUniqueWaveSpectraKumar( wave_spectra_Kumar &ws,vector<wave_spectra_Kumar> &wsList )

106 {107 int i=0;108 bool found = false;109 for( i=0; i<wsList.size(); i++ )110 {111 wave_spectra_Kumar wsTmp = wsList[i];112 if( ws.numPartitions == wsTmp.numPartitions &&113 ws.shipDirectionNorthDeg == wsTmp.shipDirectionNorthDeg &&114 ws.useCompassHeadings == wsTmp.useCompassHeadings &&115 ws.sigWaveHeight == wsTmp.sigWaveHeight &&116 ws.spreadAngleDeg == wsTmp.spreadAngleDeg &&117 ws.spreadExp == wsTmp.spreadExp &&118 ws.waveDir == wsTmp.waveDir &&119 ws.waveHeadingsDegs == wsTmp.waveHeadingsDegs &&120 ws.wavePeriod == wsTmp.wavePeriod )121 {122 found = true;123 break;124 }125 }


126

127 if( !found )128 {129 ws.wsIndex = (int)wsList.size();130 wsList.push_back( ws );131 return ws.wsIndex;132 }133 else134 {135 ws.wsIndex = i;136 return ws.wsIndex;137 }138 }139

140 double roundHeadingFunc( const double &heading, const double &roundVal )141 {142 double retval = 0.0;143 int numIntervals = int(360.0 / roundVal);144 double lowerBound = 0;145 double upperBound = lowerBound + roundVal;146

147 for( int i=0; i<numIntervals; i++ )148 {149 if( heading + roundVal/2.0 >= lowerBound && heading + roundVal/2.0 <

upperBound )150 {151 retval = lowerBound;152 break;153 }154

155 lowerBound = upperBound;156 upperBound += roundVal;157 }158 return retval;159 }160

161 double roundSpeedFunc( const double &speed, const double &roundVal)162 {163 double retval = std::floor((speed + roundVal/2.0) / roundVal) * roundVal;164 return retval;165 }166

167 void getHoodKumarWSData( PGconn* conn, double roundHeading, conststd::string &date, const double &lat, const double &lon, vector<double>


&allSigHeights, vector<double> &allPeakPeriods,168 vector<double> &allWaveDirs, vector<double> &allSpreadingExp,

vector<string> &allHindcastLats, vector<string> &allHindcastLons,wave_spectra_Kumar &ws, const double& shipDirectionNorthDeg )

169 {170 vector<double> allSpreadingAngles;171 ptime day1 (time_from_string(date));172 ptime day2 = day1 + minutes((long)90.0);173

174 std::string date_search = "abs(extract(epoch from (rec_date - timestamp '"+ date + "')))";

175 std::string date1 = to_simple_string(day1.date()) + ' ' +to_string(day1.time_of_day().hours()) + ':' +to_string(day1.time_of_day().minutes());

176 std::string date2 = to_simple_string(day2.date()) + ' ' +to_string(day2.time_of_day().hours()) + ':' +to_string(day2.time_of_day().minutes());

177

178 std::stringstream querryPatitionLocationStr;179 querryPatitionLocationStr << "SELECT lon_lat, rec_date FROM

global_partitions WHERE partition = 1 AND rec_date >= '" + date1 + "'AND rec_date < '" + date2 + "' AND ST_DISTANCE(lon_lat,ST_SetSRID(ST_Point(" + to_string(lon) + ", " + to_string(lat) +"),4326)) < 5.0 ORDER BY ST_DISTANCE(lon_lat, ST_SetSRID(ST_Point(" +to_string(lon) + ", " + to_string(lat) + "),4326)), " + date_search + "LIMIT 1";

180

181 PGresult *pgres = PQexec(conn, querryPatitionLocationStr.str().c_str());182 if(PQresultStatus(pgres) != PGRES_TUPLES_OK)183 {184 fprintf(stderr, "query failed: %s", PQerrorMessage(conn));185 exit_nicely(conn,pgres);186 }187

188 std::string lon_lat;189 std::string rec_date;190 if( PQnfields(pgres) )191 {192 lon_lat = PQgetvalue(pgres, 0, 0);193 rec_date = PQgetvalue(pgres, 0, 1);194 }195 else196 cout << "failed" << endl;197


198 PQclear(pgres);199

200 std::stringstream querryPatitionDataStr;201 querryPatitionDataStr << "SELECT partition, wave_height, peak_period,

wave_length, wave_dir, spreading_angle, wind_sea_fraction, lon, lat,rec_date FROM global_partitions WHERE rec_date = '" + rec_date + "' ANDlon_lat = '" + lon_lat + "'";

202

203 pgres = PQexec(conn, querryPatitionDataStr.str().c_str());204 if(PQresultStatus(pgres) != PGRES_TUPLES_OK)205 {206 fprintf(stderr, "query failed: %s", PQerrorMessage(conn));207 exit_nicely(conn,pgres);208 }209

210 int numRows = 0;211 if(numRows = PQntuples(pgres))212 {213 std::string *partition = new std::string[numRows];214 std::string *wave_height = new std::string[numRows];215 std::string *peak_period = new std::string[numRows];216 std::string *wave_length = new std::string[numRows];217 std::string *wave_dir = new std::string[numRows];218 std::string *spreading_angle = new std::string[numRows];219 std::string *wind_sea_fraction = new std::string[numRows];220 std::string *plon = new std::string[numRows];221 std::string *plat = new std::string[numRows];222 std::string *rec_date = new std::string[numRows];223

224 for(int i=0; i<numRows; i++)225 {226 partition[i] = PQgetvalue(pgres, i, 0);227 wave_height[i] = PQgetvalue(pgres, i, 1);228 peak_period[i] = PQgetvalue(pgres, i, 2);229 wave_length[i] = PQgetvalue(pgres, i, 3);230 wave_dir[i] = PQgetvalue(pgres, i, 4);231 spreading_angle[i] = PQgetvalue(pgres, i, 5);232 wind_sea_fraction[i] = PQgetvalue(pgres, i, 6);233 plon[i] = PQgetvalue(pgres, i, 7);234 plat[i] = PQgetvalue(pgres, i, 8);235 rec_date[i] = PQgetvalue(pgres, i, 9);236

237 allSigHeights.push_back(::atof(wave_height[i].c_str()));238 allPeakPeriods.push_back(::atof(peak_period[i].c_str()));


239 allSpreadingAngles.push_back(::atof(spreading_angle[i].c_str()));240 allWaveDirs.push_back(roundHeadingFunc(::atof(wave_dir[i].c_str()),

roundHeading));241 allSpreadingExp.push_back(

(2/pow(::atof(spreading_angle[i].c_str())*M_PI/180.0,2))-1 );242

243 ws.sigWaveHeight.push_back(::atof(wave_height[i].c_str()));244 ws.wavePeriod.push_back(::atof(peak_period[i].c_str()));245 ws.spreadAngleDeg.push_back(::atof(spreading_angle[i].c_str()));246 ws.waveDir.push_back(::atof(wave_dir[i].c_str()));247

248 int spreadExp =int((2/pow(::atof(spreading_angle[i].c_str())*M_PI/180.0,2))-1);

249 if(spreadExp > 80)250 spreadExp = 80;251 if(spreadExp < 1)252 spreadExp = 1;253

254 ws.spreadExp.push_back( spreadExp );255 }256

257 ws.numPartitions = numRows;258 ws.useCompassHeadings = true;259 ws.shipDirectionNorthDeg = shipDirectionNorthDeg;260 }261 else262 cout << "failed" << endl;263

264 PQclear(pgres);265 }266

267 bool positionSort(position i, position j) { return i.distance < j.distance;}

268

269 vector<position> find_closest_point( const double &lat, const double &lon,PGconn *conn )

270 {271 vector<std::string> lon_lat;272 double startLat = floor(lat * 2) / 2;273 double endLat = startLat + 0.5;274 if( lat < 0 )275 {276 endLat = ceil(lat * 2) / 2;277 startLat = endLat - 0.5;


278 }279

280 double startLon = startLon = floor(lon * 2) / 2;281 double endLon = startLon + 0.5;282 if( lon < 0 )283 {284 endLon = ceil(lon * 2) / 2;285 startLon = endLon - 0.5;286 }287 vector<position> positions;288

289 for( double nextLat=startLat; nextLat<=endLat; nextLat += 0.5 )290 {291 for( double nextLon=startLon; nextLon<=endLon; nextLon += 0.5 )292 {293 // calculate the distance between lat/lon and nextLat/nextLon294 position newPos;295 newPos.distance = sqrt( pow(lon-nextLon, 2) + pow(lat-nextLat, 2) );296 newPos.lat = nextLat;297 newPos.lon = nextLon;298

299 if( newPos.distance < sqrt(2.0) )300 positions.push_back( newPos );301 }302 }303

304 sort( positions.begin(), positions.end(), positionSort );305

306 return positions;307 }308

309 void getHoodBretschneiderWSData( PGconn* conn, double roundHeading,std::string &date, const double &lat, const double &lon, vector<double>&allSigHeights,

310 vector<double> &allPeakPeriods, vector<double> &allWaveDirs, vector<string>&allHindcastLats, vector<string> &allHindcastLons,vector<wave_spectra_Bretschneider> &wsList, wave_spectra_Bretschneider&ws, string& dbName )

311 {312 bool found = false;313 vector<position> positions = find_closest_point( lat, lon, conn );314

315 for(int i=0; i<positions.size(); i++)316 {


317 string lon_lat;318 std::stringstream querryStr1;319 querryStr1 << "SELECT ST_GeomFromText('POINT(" +

to_string(positions[i].lon) + " " + to_string(positions[i].lat) +")',4326)";

320

321 PGresult *pgres1 = PQexec(conn, querryStr1.str().c_str());322 if(PQresultStatus(pgres1) != PGRES_TUPLES_OK)323 {324 fprintf(stderr, "query failed: %s", PQerrorMessage(conn));325 exit_nicely(conn,pgres1);326 }327

328 if( PQntuples(pgres1) )329 lon_lat = PQgetvalue(pgres1, 0, 0);330

331 // extract the year from the date string332 int year = 0;333 std::vector<std::string> year_month_day = split( date, '-');334 year = atoi( year_month_day[0].c_str() );335

336 std::stringstream querryStr;337 querryStr << "SELECT wave_height, wave_dir, wave_period, lat, lon\nFROM \""

<< dbName << year << "\"\nWHERE rec_date = '" + date + "'\nAND lon_lat= '" + lon_lat + "'";

338

339 PGresult *pgres = PQexec(conn, querryStr.str().c_str());340 if(PQresultStatus(pgres) != PGRES_TUPLES_OK)341 {342 fprintf(stderr, "query failed: %s", PQerrorMessage(conn));343 exit_nicely(conn,pgres);344 }345

346 std::string height;347 std::string direction;348 std::string period;349 std::string hclat;350 std::string hclon;351 if( PQntuples(pgres) )352 {353 height = PQgetvalue(pgres, 0, 0);354 direction = PQgetvalue(pgres, 0, 1);355 period = PQgetvalue(pgres, 0, 2);356 hclat = PQgetvalue(pgres, 0, 3);


357 hclon = PQgetvalue(pgres, 0, 4);358

359 if(height == "" || period == "")360 break;361

362 allSigHeights.push_back(::atof(height.c_str()));363 allPeakPeriods.push_back(::atof(period.c_str()));364 allWaveDirs.push_back(roundHeadingFunc(::atof(direction.c_str()),

roundHeading));365 allHindcastLats.push_back(hclat);366 allHindcastLons.push_back(hclon);367

368 ws.hSig = ::atof(height.c_str());369 ws.wavePeakPeriod = ::atof(period.c_str());370 found = true;371 break;372 }373 }374

375 if( !found )376 {377 allSigHeights.push_back( -1 );378 allPeakPeriods.push_back( -1 );379 allWaveDirs.push_back( -1 );380 allHindcastLats.push_back( "NULL" );381 allHindcastLons.push_back( "NULL" );382 }383 }384

385 int main(int argc, char* argv[])386 {387 clock_t end_time, first_time;388 int err = 0;389 char conninfo[250];390 string line;391 PGconn *conn = NULL;392 PGresult *pgres = NULL;393 string dbName;394

395 std::string prevDate = "";396 double prevLat = -1;397 double prevLon = -1;398

399 vector<wave_spectra_Bretschneider> wsList;


400 vector<wave_spectra_Kumar> wsKumarList;401 vector<double> sigHeights;402 vector<double> peakPeriods;403 vector<double> speeds;404 vector<double> headings;405 vector<double> allSigHeights;406 vector<double> allPeakPeriods;407 vector<double> allSpeeds;408 vector<double> allHeadings;409 vector<double> allWaveDirs;410 vector<double> allSpreadingExp;411 vector<ptime> allDates;412 vector<double> allLats;413 vector<double> allLons;414 vector<int> allWSIndices;415 vector<double> allProbabilities;416 vector<string> allHindcastLats;417 vector<string> allHindcastLons;418

419 first_time = clock(); // Start time for reading the data from the cdf file420

421 if (argc != 6) usage(argv[0]);422

423 const char* filename = _strdup(argv[1]);424 const char* wsType = _strdup(argv[2]);425 double roundHeading = atof(argv[3]);426 double roundSpeed = atof(argv[4]);427 const char* dbType = _strdup(argv[5]);428

429 if( strcmp(dbType, "WWIII") == 0 )430 dbName = "global_wwiii_data_";431 else if( strcmp(dbType, "ERA5") == 0 )432 dbName = "global_ERA5_data_";433 else434 {435 cout << "Invalid database type.";436 exit(1);437 }438

439 ifstream gpsDataFile( filename );440 if( !gpsDataFile.is_open() )441 {442 printf("ERROR: unable to open file %s\n",filename);443 return 1;


444 }445




457 // Read past the header458 getline( gpsDataFile, line );459

460 std::string::size_type sz;461 bool first = true;462 int year = 0;463 int month = 0;464 int day = 0;465 int hour = 0;466 int minute = 0;467

468 // Format the date output in day/month/year hour:minute format469 boost::posix_time::time_facet* facet = new boost::posix_time::time_facet();470 facet->format("%d/%m/%Y %H:%M");471 cout.imbue(std::locale(std::locale::classic(), facet));472

473 while ( getline( gpsDataFile, line ) )474 {475 std::vector<std::string> columns = split( line, '\t');476

477 if(columns.size() < 6 || columns[4] == "")478 continue;479

480 std::vector<std::string> date_time = split( columns[0], ' ');481 std::vector<std::string> day_month_year = split( date_time[0], '/');482 std::vector<std::string> hours_minutes = split( date_time[1], ':');483

484 year = std::stoi( day_month_year[2], &sz );485 month = std::stoi( day_month_year[1], &sz );486 day = std::stoi( day_month_year[0], &sz );


487 hour = std::stoi( hours_minutes[0], &sz );488 minute = std::stoi( hours_minutes[1], &sz );489

490 if( minute >= 30 )491 {492 hour++;493 if( hour > 24 )494 hour = 24;495 }496

497 if( hour < 24.0 && hour >= 22.5 )498 hour = 24;499 else if( hour < 22.5 && hour >= 21.0 )500 hour = 21;501 else if( hour < 21.0 && hour >= 19.5 )502 hour = 21;503 else if( hour < 19.5 && hour >= 18.0 )504 hour = 18;505 else if( hour < 18.0 && hour >= 16.5 )506 hour = 18;507 else if( hour < 16.5 && hour >= 15.0 )508 hour = 15;509 else if( hour < 15.0 && hour >= 13.5 )510 hour = 15;511 else if( hour < 13.5 && hour >= 12.0 )512 hour = 12;513 else if( hour < 12.0 && hour >= 10.5 )514 hour = 12;515 else if( hour < 10.5 && hour >= 9.0 )516 hour = 9;517 else if( hour < 9.0 && hour >= 7.5 )518 hour = 9;519 else if( hour < 7.5 && hour >= 6.0 )520 hour = 6;521 else if( hour < 6.0 && hour >= 4.5 )522 hour = 6;523 else if( hour < 4.5 && hour >= 3.0 )524 hour = 3;525 else if( hour < 3.0 && hour >= 1.5 )526 hour = 3;527 else if( hour < 1.5 && hour >= 0.0 )528 hour = 0;529

530 double lat = std::stod(columns[1]);


531 double lon = std::stod(columns[2]);532

533 if( lon > 180.0 )534 lon = lon - 360.0;535

536 ptime st(date(year,month,day), hours(hour)+minutes(0));537 std::string date = to_string(year) + '-' + to_string(month) + '-' +

to_string(day) + ' ' + to_string(hour) + ":0:0";538

539 double waveDir = 0.0;540 if( strcmp(wsType, "1D") == 0 )541 {542 if( !(prevDate.compare(date) == 0 && lat == prevLat && lon == prevLon) )543 {544 // if the date and/or position have changed then retrieve new hindcast data545 wave_spectra_Bretschneider ws;546 getHoodBretschneiderWSData( conn, roundHeading, date, lat, lon,

allSigHeights, allPeakPeriods, allWaveDirs, allHindcastLats,allHindcastLons, wsList, ws, dbName );

547 allWSIndices.push_back( insertUniqueWaveSpectra( ws, wsList ) );548 }549 else550 {551 // if the date and position are the same then repeat the last entry552 size_t i = allSigHeights.size()-1;553 allSigHeights.push_back( allSigHeights[i] );554 allPeakPeriods.push_back( allPeakPeriods[i] );555 allWaveDirs.push_back( allWaveDirs[i] );556 allHindcastLats.push_back( allHindcastLats[i] );557 allHindcastLons.push_back( allHindcastLons[i] );558 allWSIndices.push_back( allWSIndices[i] );559 }560 prevDate = date;561 prevLat = lat;562 prevLon = lon;563 }564 else if( strcmp(wsType, "2D") == 0 )565 {566 wave_spectra_Kumar wsKumar;567 getHoodKumarWSData( conn, roundHeading, date, lat, lon, allSigHeights,

allPeakPeriods, allWaveDirs, allSpreadingExp, allHindcastLats,allHindcastLons, wsKumar, ::atof(columns[4].c_str()) );

568 }569 else


570 {571 cout << "wsType not recognized." << endl;572 return(0);573 }574

575 allDates.push_back(st);576 allLats.push_back(lat);577 allLons.push_back(lon);578 allSpeeds.push_back(roundSpeedFunc(::atof(columns[3].c_str()), roundSpeed));579

580 double cog = 0.0;581 double heading = ::atof(columns[4].c_str());582 if( heading + 180.0 >= 360 )583 cog = heading - 180.0;584 else585 cog = heading + 180.0;586

587 double relAng = 0.0;588 if( cog - waveDir >= 0.0 )589 relAng = cog - waveDir;590 else591 relAng = 360.0 + cog - waveDir;592 double relHeading = roundHeadingFunc(relAng, roundHeading);593 allHeadings.push_back( relHeading );594 }595

596 // Close the GPS file597 gpsDataFile.close();598

599 if( strcmp(wsType, "1D") == 0 )600 {601 cout << "Date lat lon speed heading height wave dir peak period ws index

hc lat hc lon\n";602 for(int i=0; i<allDates.size(); i++)603 {604 cout << allDates[i] << '\t' << allLats[i] << '\t' << allLons[i] << '\t' <<

allSpeeds[i] << '\t' << allHeadings[i] << '\t' << allSigHeights[i] <<'\t' << allWaveDirs[i] << '\t' << allPeakPeriods[i] << '\t' <<allWSIndices[i] << '\t' << allHindcastLats[i] << '\t' <<allHindcastLons[i] << endl;

605 }606 }607

608 /* close the connection to the database and cleanup */


609 PQfinish(conn);610

611 end_time = clock() - first_time;612 printf ("Total time = %f minutes\n",((float)end_time)/CLOCKS_PER_SEC/60.0);613 printf("*** SUCCESS reading file: %s\n", filename);614

615 return 0;616 }


DOCUMENT CONTROL DATA*Security markings for the title, authors, abstract and keywords must be entered when the document is sensitive

1. ORIGINATOR (Name and address of the organization preparing thedocument. A DRDC Centre sponsoring a contractor’s report, or atasking agency, is entered in Section 8.)

DRDC – Atlantic Research CentrePO Box 1012, Dartmouth NS B2Y 3Z7, Canada

2a. SECURITY MARKING (Overall security marking ofthe document, including supplemental markings ifapplicable.)

CAN UNCLASSIFIED

2b. CONTROLLED GOODS

NON-CONTROLLED GOODSDMC A

3. TITLE (The document title and sub-title as indicated on the title page.)

Accessing wave hindcast data

4. AUTHORS (Last name, followed by initials – ranks, titles, etc. not to be used. Use semi-colon as delimiter)

Thompson, I.; Nickerson, J.

5. DATE OF PUBLICATION (Month and year of publication ofdocument.)

July 2019

6a. NO. OF PAGES (Totalpages, including Annexes,excluding DCD, coveringand verso pages.)

45

6b. NO. OF REFS (Totalcited in document.)

10

7. DOCUMENT CATEGORY (e.g., Scientific Report, Contract Report, Scientific Letter)

Reference Document

8. SPONSORING CENTRE (The name and address of the department project or laboratory sponsoring the research anddevelopment.)

DRDC – Atlantic Research CentrePO Box 1012, Dartmouth NS B2Y 3Z7, Canada

9a. PROJECT OR GRANT NO. (If appropriate, the applicableresearch and development project or grant number underwhich the document was written. Please specify whetherproject or grant.)

01ef

9b. CONTRACT NO. (If appropriate, the applicable contractnumber under which the document was written.)

10a. DRDC DOCUMENT NUMBER

DRDC-RDDC-2019-D10510b. OTHER DOCUMENT NO(s). (Any other numbers which may

be assigned this document either by the originator or by thesponsor.)

11a. FUTURE DISTRIBUTION WITHIN CANADA (Approval for further dissemination of the document. Security classification must alsobe considered.)

Public release

11b. FUTURE DISTRIBUTION OUTSIDE CANADA (Approval for further dissemination of the document. Security classification must alsobe considered.)

Public release

12. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Use semi-colon as a delimiter.)

wave hindcasts; virtual hull monitoring; hull monitoring, fatigue

13. ABSTRACT/RÉSUMÉ (When available in the document, the French version of the abstract must be included here.)

A procedure to gain access to wave data from large wave hindcast datasets is described. Thisincludes explaining how to create a database, how to populate the database with wave datain two binary formats commonly used for these data, and how to extract information from thedatabase. This procedure will make it easier for readers to access freely-available hindcast wavedata. Although the procedure has been developed to access NOAA and ECMWF hindcasts, itmay be suitable for other hindcasts that use the same data formats.

On décrit une procédure d’accès à de l’information sur les vagues provenant de prévisions a pos-teriori réalisées à partir de grands ensembles de données. On explique, notamment, commentcréer une base de données, comment l’alimenter avec des données sur les vagues dans deuxformats binaires d’usage courant pour ce type de données et comment extraire de l’information.Cette procédure aidera les lecteurs qui veulent utiliser les données à accès libre sur le sujet.Même si celle-ci a été élaborée spécifiquement pour faciliter l’accès aux prévisions de la NationalOceanic and Atmospheric Administration (NOAA) des États-Unis et à celles du Centre européenpour les prévisions météorologiques à moyen terme (CEPMMT), elle pourrait tout de même êtreutile dans le cas d’autres prévisions comportant des données dans les mêmes formats.

accessing wave hindcast data - defence research and ... · 1 d:\exe_folder\loadgrib...

Documents