use of openearthtools to make netcdf 1 3 d:\... 3 http://… tools models add meta information...

Use of OpenEarthTools to make netCDF

1

3 D:\...

3 http://…

tools

models

add meta information

netCDF on web server

transform to netCDF

netCDF on OPeNDAP

server

data

data providers = data users

Extract Transform Load Provide

Subversion repository

2

Contents

1

3 D:\...

3 http://…

tools

models

add meta information

netCDF on web server

transform to netCDF

netCDF on OPeNDAP

server

data

data providers = data users

2a

use tools >

Extract Transform Load Provide

2b

read data

2c

> save as netCDF

OpenEarthTools

• Download and install TortoiseSVN Subversion clienthttp://subversion.tigris.org/

• Make a checkout of OpenEarthTools (full or just Matlab subset)• Start Matlab and go to OpenEarthTools

>> cd F:\checkouts\OpenEarthTools\matlab\• Add OpenEarth paths to Matlab:

>> oetsettings• Inspect OpenEarthTools• When many people work on oen thing, we need a few basic rules• Document your functions and toolboxes

• Exploit the clickable links to files + directories in new Matlab version>> help functionname % in path>> help directoryname % in path• Displays:

> all H1 lines (first comment line in file)> or displays help in contents.m

• Add crosslinks: with 3 crosslinks, 6 clicks connect 729 files ≈ size OpenEarthTools• See also

OpenEarthTools documentation

other directories in path

other m function

www links

>> help applications

OpenEarth applications rijkswaterstaat - data: rijkswaterstaat data types (donar) delft3d - delftd file formats knmi - read knmi wind and meteo timeseries swan - swan toolbox

>> help rijkswaterstaat

donar_read - read ASCII text file from www.waterbase.nlgetwaterbase - get data from waterbase.nlgetwaterbasestation - get stations names for parametergetwaterbaseparameter - get parameter from waterbase.nlhmcz_read - read meteo file from hmcz@ rws

>> help donar_read

DONAR_READ read ASCII text file from www.waterbase.nl

DAT = donar_read(fname,<keyword,value>)

See also: getwaterbase, www.waterbase.nl

% DONAR_READ read ASCII text file from www.waterbase.nl%% DAT = donar_read(fname,<keyword,value>)%% See also: getwaterbase, www.waterbase.nl

OpenEarthTools documentation

H1 line

contents.m

donar_read.m

Contents

1

3 D:\...

3 http://…

tools

models

add meta information

netCDF on web server

transform to netCDF

netCDF on OPeNDAP

server

data

data providers = data users

Extract Transform Load Provide

2a

use tools >

2b

read data

2c

> save as netCDF

Read a raw dataset

• Always read data into one struct

• Separate columns (Excel) into fieldnames• Include as much meta-info as possible in the struct• Use English names• Use SI units• Use documentation: cells are block starting with %% (~ chapters)• Align

Read a raw dataset

• Dedicated matlab tools>> xlsread>> textscan>> csvread

• Dedicated OpenEarthTools• www.waterbase.nl

>> donar_read• http://www.knmi.nl/klimatologie/onderzoeksgegevens/potentiele_wind/

>> KNMI knmi_potwind• http://www.knmi.nl/klimatologie/daggegevens/download.html

>> knmi_etmgeg• www.NODC.nl / SeaDataNet profile data (CTD):

>> odvread• Dedicate yourselves by peeling the file & add to OpenEarthTools:

>> oetnewfun, load_template>> fgetl, fscanf % ascii>> fread % binary

Read a raw data waterbase

Read a raw data KNMI wind

Read a raw data KNMI meteo

Read a raw data NODC ODV

Test with some raw data

Test for yourselve some raw data with dedicated scripts, choose one.Find the associated matlab function using the help documentation.

• Rijkswaterstaatwww.waterbase.nl

• KNMIhttp://www.knmi.nl/klimatologie/onderzoeksgegevens/

potentiele_wind/http://www.knmi.nl/klimatologie/daggegevens/download.html

• NODCwww.NODC.nl / SeaDataNet profile data (CTD):

Inspect a raw dataset

• Different types of raw data• Different structs for time series. Hmm.

> Waterbase > KNMI wind> KNMI meteo

• How to unify?• No guidelines for structs• International guidelines for netCDF!

• Idea: collaborate!• Each BwN participant transforms

raw data they use anyway into netCDF

• Everyone can readily access previously used data in a uniform manner

raw data struct

raw data file

netCDF

No standards available

International standards available

Contents

1

3 D:\...

3 http://…

tools

models

add meta information

netCDF on web server

transform to netCDF

netCDF on OPeNDAP

server

data

data providers = data users

Extract Transform Load Provide

2a

use tools >

2b

read data

2c

> save as netCDF

netCDF

• What is netCDF• How to make netCDF

• Matlab netCDF interface

• Conventions (vocabularies):> Standard meta-info: CF convention +

GNU/CC/?> standard quantities: CF convention> Standard units: Unidata UD units> Standard coordinate codes: EPSG

• Save script in repository

• OpenEarthRawData: for dedicated script

• OpenEarthTools: for more general script• Inspect and plot netCDF

netCDF = linked dimensions, variables, attributes

t =f(t)z = f(z)

# = f(#)

chlorophyll = f(t,z,#)

disclaimer

institute

copyright

long_name

standard_name

units

long_name

standard_name

units

long_name

units

long_name

standard_name

unitsconventions

title

website

t

#

z

Self-describing file:• meta-info

never lost• no manual

needed• open file format

guarantees

future access

Let’s look at a basic netCDF file

nc_cf_example2nc.nc (in OpenEarthTools checkout)• >> nc_dump(‘nc_cf_example2nc.nc’)• ncBrowse • HDFview

Hierarchical structure as file system• Attributes• Dimensions• Variables

netCDF (1)

• Any amount of meta-data (Attributes) can be appended. • Handles large datasets (file > 4GB allowed with in 64-bit offset mode). • Scaleable for future applications (divide data over multiple files). • Open source specifications and interfaces.• Fast data storage and retrieval (direct access binary). • Small data storage (binary is small + internal zip option). • Simple yet robust.

“Following examination by 20 reviewers, NASA's Earth Science Data Systems Standards Process Group has concluded that netCDF classic should be adopted as a recommended standard.A major strength of netCDF classic, according to the reviewers, was that it has fostered data interoperability and exchange through its self-describing file format, platform independent architecture, and robust access methods. Additionally, its overall file format and metadata attributes were simple enough to be easily understood and applied yet robust enough to describe and store multidimensional data of different types in the same file” (source: Unidata news)

netCDF (2)

• 2 independent implementations of NetCDF interface (C & Java):

• reliable

• redundant

• Generally used, so there’s always someone who can offer help available.

• netCDF is de facto standard for coastal ocean circulation models (GOTM, ROMS, ECOM-SED and in the near future also Delft3D-FLOW).

• Lots of simple and free tools on the web that operate on NetCDF files:

• ncBrowse (NOAA)

• NCO

• Panoply (NASA)

• ncview (linux)

• Implemented in numerous commercial tools too

• Quickplot (Delft3D)

netCDF (3)

• Interfaces to sophisticated scientific data manipulation languages: • C-interface• Java-interface• Fortran• Python• IDL• Matlab

> Through java-interface (in OET)> mexcdf for C-interface (in OET)> Native matlab interface (since release 2008b)

• Merged with OPeNDAP-interface• Handle netCDF file on web as if it on your own PC.

• Additional, widely accepted conventions for NetCDF• implemented in main NetCDF viewers • CF meta data convention

Creating a netCDF file

• See : nc_cf_example2nc.m• Now step by step walking through• Use nc_cf_example2nc.nc as template when doing it yourselves

>> nc_create_empty % 1

disclaimer

institute

copyright

long_name

standard_name

units

long_name

standard_name

units

long_name

units

long_name

standard_name

unitsconventions

title

website

t

#

z

t =f(t)z = f(z)

# = f(#)

chlorophyll = f(t,z,#)

>> nc_create_empty % 0

• Web-sources (waterbase, knmi)

• Read directly from web

• Or download to local cache• Your own data:

• Save raw data in repository

OpenEarthRawData with

• INSPIRE XML meta-data• Then read data in Matlab• and create empty netCDF file

>> nc_attput % 1

disclaimer

institute

copyright

long_name

standard_name

units

long_name

standard_name

units

long_name

units

long_name

standard_name

unitsconventions

title

website

t

#

z

t =f(t)z = f(z)

# = f(#)

chlorophyll = f(t,z,#)

>> nc_attput % 1

• See CF meta-data conventions: http://cf-pcmdi.llnl.gov/• OpenEarth conventions:

• disclaimer + terms_for_use

• $Headurl$

>> nc_add_dimension % 2

disclaimer

institute

copyright

long_namestandard_name

units

long_name

standard_name

units

long_name

units

long_name

standard_name

unitsconventions

title

website

t

#

z

chlorophyll = f(t,z,#)

t =f(t)z = f(z)

# = f(#)

>> nc_add_dimension % 2

>> nc_addvar % 3

disclaimer

institute

copyright

long_name

standard_name

units

long_name

standard_name

units

long_name

units

long_name

standard_name

unitsconventions

title

website

t

#

z

chlorophyll = f(t,z,#)

t =f(t)z = f(z)

# = f(#)

>> nc_addvar % 3

CF convention

vocabulary

of

quantities

>> nc_varput % 4

>> nc_varput % 4

UNIDATA UD units package

vocabulary

of

units

>> nc_varput % 4

disclaimer

institute

copyright

long_name

standard_name

units

long_name

standard_name

units

long_name

units

long_name

standard_name

unitsconventions

title

website

t

#

z

chlorophyll = f(t,z,#)

t =f(t)z = f(z)

# = f(#)

>> nc_varput % 4

>> nc_dump % 5

disclaimer

institute

copyright

conventions

title

website

t

#

long_name

standard_name

units

z

long_namestandard_name

units

long_name

standard_name

units

long_name

units

chlorophyll = f(t,z,#)

t =f(t)z = f(z)

# = f(#)

>> nc_dump % 5

>> nc_dump % 5

Plot netCDF in free ncBrowse

Plot netCDF in free ncBrowse

Plot netCDF in Matlab

Plot netCDF in ArcGis

ArcGis also reads and writes netcdf files.

ArcGis also reads and writes netcdf files.

ArcGIS

Check netCDF in your own text editor (xml)

xml representation of a netcdf file

xml representation of a netcdf file

XML

Check netCDF in …

NCO#diffncdiff -v time file1.nc file2.nc#compression & packingncpdq -4 -L 9 in.nc out.nc # Deflated packing (~80% lossy compression)#selecting variables by regexncks -v '^Q..' in.nc # Q01--Q99, QAA--QZZ, etc.

IDVVery usefulVery useful

Web hyperslabs, cool!Web hyperslabs, cool!

Not so stable.Not so stable.

JAVA netCDF toolsUsefulUseful

Summary current session

1

3 D:\...

3 http://…

tools

models

add meta information

netCDF on web server

transform to netCDF

netCDF on OPeNDAP

server

data

data providers = data users

2a

use tools >

Extract Transform Load Provide

2b

read data

2c

> save as netCDF

Next session: using netCDF from others

1

3 D:\...

3 http://…

tools

models

add meta information

netCDF on web server

transform to netCDF

netCDF on OPeNDAP

server

data

data providers = data users

Extract Transform Load Provide

2